Sep 06 13:55:53.956063 ip-10-0-132-113 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 06 13:55:53.956074 ip-10-0-132-113 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 06 13:55:53.956084 ip-10-0-132-113 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 06 13:55:53.956431 ip-10-0-132-113 systemd[1]: Failed to start Kubernetes Kubelet. Sep 06 13:56:02.673480 ip-10-0-132-113 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Sep 06 13:56:02.673496 ip-10-0-132-113 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ffadcbcec5af4e74bf1634e6e89f82a4 -- Sep 06 13:57:48.399172 ip-10-0-132-113 systemd[1]: Starting Kubernetes Kubelet... Sep 06 13:57:48.873431 ip-10-0-132-113 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 06 13:57:48.873431 ip-10-0-132-113 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 06 13:57:48.873431 ip-10-0-132-113 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 06 13:57:48.873431 ip-10-0-132-113 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 06 13:57:48.873431 ip-10-0-132-113 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 06 13:57:48.874079 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.873521 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 06 13:57:48.876489 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876473 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 13:57:48.876489 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876488 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876492 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876495 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876498 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876502 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876505 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876507 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876510 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876514 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876517 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876520 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876523 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876525 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876528 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876531 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876533 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876536 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876539 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876541 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876544 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 13:57:48.876552 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876546 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876549 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876551 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876554 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876557 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876560 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876566 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876569 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876572 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876574 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876577 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876579 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876582 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876584 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876587 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876590 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876592 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876594 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876597 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 13:57:48.877035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876599 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876603 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876607 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876610 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876612 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876615 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876617 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876620 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876622 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876625 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876627 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876630 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876632 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876635 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876638 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876642 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876645 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876647 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876650 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 13:57:48.877571 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876652 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876655 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876657 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876660 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876662 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876665 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876668 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876671 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876673 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876676 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876680 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876684 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876687 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876690 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876693 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876695 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876698 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876700 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876703 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 13:57:48.878042 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876705 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876708 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876711 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876714 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876719 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876721 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876725 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876728 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876732 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876735 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876738 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876741 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876743 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876746 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876749 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876752 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876754 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876757 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876761 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876764 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 13:57:48.878499 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.876766 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877459 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877466 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877469 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877472 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877475 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877477 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877480 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877483 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877485 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877488 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877491 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877493 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877496 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877499 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877503 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877507 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877511 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877514 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 13:57:48.878996 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877517 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877520 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877523 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877525 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877528 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877531 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877533 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877536 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877538 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877541 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877543 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877545 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877548 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877550 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877553 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877555 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877558 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877560 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877563 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877566 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 13:57:48.879465 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877568 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877570 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877573 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877576 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877579 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877582 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877586 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877589 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877592 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877596 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877599 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877602 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877605 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877608 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877611 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877614 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877617 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877619 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877622 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 13:57:48.879964 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877624 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877627 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877629 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877632 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877634 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877638 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877640 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877643 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877645 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877648 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877651 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877653 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877656 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877658 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877661 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877663 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877666 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877669 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877673 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877675 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 13:57:48.880422 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877678 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877680 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877683 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877689 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877692 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877694 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877697 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877700 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877703 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877705 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877708 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877711 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877713 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877716 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877718 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877721 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877723 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877726 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877729 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877731 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 13:57:48.880918 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877734 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.877737 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878107 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878112 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878115 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878118 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878121 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878124 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878127 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878129 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878133 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878135 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878138 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878140 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878143 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878146 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878149 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878151 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878154 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 13:57:48.881402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878157 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878159 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878162 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878166 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878170 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878173 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878176 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878179 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878182 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878184 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878187 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878190 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878193 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878196 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878198 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878201 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878205 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878208 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878212 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 13:57:48.881857 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878215 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878218 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878220 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878223 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878226 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878228 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878231 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878234 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878238 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878241 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878244 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878247 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878250 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878252 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878255 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878257 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878260 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878262 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878265 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878267 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 13:57:48.882366 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878270 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878272 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878275 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878277 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878280 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878282 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878285 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878288 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878290 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878293 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878296 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878298 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878300 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878303 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878306 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878308 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878311 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878313 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878316 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878318 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 06 13:57:48.882841 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878320 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878323 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878325 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878330 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878332 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878335 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878337 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878340 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878342 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878344 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878347 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878349 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878352 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878354 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878357 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878359 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878362 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878365 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878368 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878370 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 13:57:48.883374 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878373 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878375 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.878378 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878445 2577 flags.go:64] FLAG: --address="0.0.0.0" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878481 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878487 2577 flags.go:64] FLAG: --anonymous-auth="true" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878491 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878496 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878499 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878503 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878507 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878511 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878514 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878517 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878520 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878523 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878527 2577 flags.go:64] FLAG: --cgroup-root="" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878530 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878533 2577 flags.go:64] FLAG: --client-ca-file="" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878536 2577 flags.go:64] FLAG: --cloud-provider="external" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878539 2577 flags.go:64] FLAG: --cluster-dns="[]" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878543 2577 flags.go:64] FLAG: --cluster-domain="" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878546 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 06 13:57:48.883889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878549 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878552 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878555 2577 flags.go:64] FLAG: --container-log-max-files="5" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878558 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878561 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878564 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878567 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878570 2577 flags.go:64] FLAG: --contention-profiling="false" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878573 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878578 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878581 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878586 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878590 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878593 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878595 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878598 2577 flags.go:64] FLAG: --enable-load-reader="false" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878601 2577 flags.go:64] FLAG: --enable-server="true" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878604 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878609 2577 flags.go:64] FLAG: --event-burst="100" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878612 2577 flags.go:64] FLAG: --event-qps="50" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878615 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878618 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878621 2577 flags.go:64] FLAG: --eviction-hard="" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878625 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 06 13:57:48.884454 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878627 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878630 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878634 2577 flags.go:64] FLAG: --eviction-soft="" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878637 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878640 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878643 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878645 2577 flags.go:64] FLAG: --experimental-mounter-path="" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878648 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878651 2577 flags.go:64] FLAG: --fail-swap-on="true" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878654 2577 flags.go:64] FLAG: --feature-gates="" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878658 2577 flags.go:64] FLAG: --file-check-frequency="20s" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878661 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878664 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878667 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878670 2577 flags.go:64] FLAG: --healthz-port="10248" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878672 2577 flags.go:64] FLAG: --help="false" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878675 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-132-113.ec2.internal" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878683 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878686 2577 flags.go:64] FLAG: --http-check-frequency="20s" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878690 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878693 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878697 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878699 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 06 13:57:48.885034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878702 2577 flags.go:64] FLAG: --image-service-endpoint="" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878705 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878708 2577 flags.go:64] FLAG: --kube-api-burst="100" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878711 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878714 2577 flags.go:64] FLAG: --kube-api-qps="50" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878717 2577 flags.go:64] FLAG: --kube-reserved="" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878720 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878722 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878726 2577 flags.go:64] FLAG: --kubelet-cgroups="" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878728 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878731 2577 flags.go:64] FLAG: --lock-file="" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878734 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878737 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878740 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878745 2577 flags.go:64] FLAG: --log-json-split-stream="false" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878748 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878751 2577 flags.go:64] FLAG: --log-text-split-stream="false" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878754 2577 flags.go:64] FLAG: --logging-format="text" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878756 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878760 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878762 2577 flags.go:64] FLAG: --manifest-url="" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878765 2577 flags.go:64] FLAG: --manifest-url-header="" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878769 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878772 2577 flags.go:64] FLAG: --max-open-files="1000000" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878776 2577 flags.go:64] FLAG: --max-pods="110" Sep 06 13:57:48.885626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878779 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878783 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878785 2577 flags.go:64] FLAG: --memory-manager-policy="None" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878789 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878792 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878795 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878798 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878804 2577 flags.go:64] FLAG: --node-status-max-images="50" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878807 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878810 2577 flags.go:64] FLAG: --oom-score-adj="-999" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878813 2577 flags.go:64] FLAG: --pod-cidr="" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878816 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878821 2577 flags.go:64] FLAG: --pod-manifest-path="" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878824 2577 flags.go:64] FLAG: --pod-max-pids="-1" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878827 2577 flags.go:64] FLAG: --pods-per-core="0" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878830 2577 flags.go:64] FLAG: --port="10250" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878833 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878836 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ff75275b186ca7de" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878839 2577 flags.go:64] FLAG: --qos-reserved="" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878842 2577 flags.go:64] FLAG: --read-only-port="10255" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878845 2577 flags.go:64] FLAG: --register-node="true" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878848 2577 flags.go:64] FLAG: --register-with-taints="" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878852 2577 flags.go:64] FLAG: --registry-burst="10" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878854 2577 flags.go:64] FLAG: --registry-qps="5" Sep 06 13:57:48.886274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878857 2577 flags.go:64] FLAG: --reserved-cpus="" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878860 2577 flags.go:64] FLAG: --reserved-memory="" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878864 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878867 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878869 2577 flags.go:64] FLAG: --rotate-certificates="false" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878872 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878875 2577 flags.go:64] FLAG: --runonce="false" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878878 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878881 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878885 2577 flags.go:64] FLAG: --seccomp-default="false" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878888 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878891 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878896 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878899 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878902 2577 flags.go:64] FLAG: --storage-driver-password="root" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878919 2577 flags.go:64] FLAG: --storage-driver-secure="false" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878922 2577 flags.go:64] FLAG: --storage-driver-table="stats" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878925 2577 flags.go:64] FLAG: --storage-driver-user="root" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878928 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878931 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878934 2577 flags.go:64] FLAG: --system-cgroups="" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878938 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878943 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878946 2577 flags.go:64] FLAG: --tls-cert-file="" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878948 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 06 13:57:48.886863 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878952 2577 flags.go:64] FLAG: --tls-min-version="" Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878955 2577 flags.go:64] FLAG: --tls-private-key-file="" Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878958 2577 flags.go:64] FLAG: --topology-manager-policy="none" Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878961 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878964 2577 flags.go:64] FLAG: --topology-manager-scope="container" Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878967 2577 flags.go:64] FLAG: --v="2" Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878971 2577 flags.go:64] FLAG: --version="false" Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878975 2577 flags.go:64] FLAG: --vmodule="" Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878978 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.878982 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879130 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879134 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879138 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879141 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879144 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879146 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879151 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879154 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879156 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879162 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879166 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 13:57:48.887521 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879169 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879172 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879174 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879177 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879180 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879182 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879185 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879188 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879190 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879193 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879196 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879198 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879201 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879203 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879206 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879209 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879212 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879215 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879217 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 13:57:48.888050 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879221 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879225 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879228 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879231 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879234 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879237 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879239 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879242 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879246 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879249 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879251 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879255 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879258 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879261 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879263 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879266 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879269 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879271 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879274 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879276 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 13:57:48.888509 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879279 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879282 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879284 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879287 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879290 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879292 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879295 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879297 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879300 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879303 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879306 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879308 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879311 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879313 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879316 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879319 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879321 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879324 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879327 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879329 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 13:57:48.889002 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879334 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879336 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879339 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879343 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879346 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879348 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879351 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879354 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879356 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879359 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879362 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879365 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879369 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879372 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879375 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879378 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879381 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879384 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879387 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879390 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 13:57:48.889473 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879392 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 13:57:48.889980 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879395 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 13:57:48.889980 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879398 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 13:57:48.889980 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879401 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 13:57:48.889980 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879404 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 13:57:48.889980 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879407 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 13:57:48.889980 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879410 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 13:57:48.889980 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879424 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 13:57:48.889980 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.879428 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 13:57:48.889980 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.879989 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 06 13:57:48.889980 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.887979 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 06 13:57:48.889980 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.887997 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 06 13:57:48.889980 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.888880 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 06 13:57:48.889980 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.888888 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 06 13:57:48.889980 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.888964 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 13:57:48.889980 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.888970 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.888973 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.888976 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.888979 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.888981 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.888984 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.888987 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.888989 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.888992 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.888995 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.888997 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889000 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889003 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889006 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889008 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889013 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889017 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889020 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889024 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 13:57:48.890348 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889027 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889031 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889034 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889037 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889040 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889043 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889046 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889049 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889051 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889062 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889065 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889068 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889071 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889073 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889076 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889079 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889081 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889084 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889087 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889089 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 13:57:48.890807 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889091 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889094 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889096 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889099 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889102 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889104 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889107 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889109 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889112 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889114 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889117 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889119 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889122 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889125 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889127 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889130 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889132 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889135 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889137 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 13:57:48.891354 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889140 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889144 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889148 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889159 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889162 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889164 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889167 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889169 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889172 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889175 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889177 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889180 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889182 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889185 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889188 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889190 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889193 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889195 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889198 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 13:57:48.891816 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889200 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889203 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889205 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889208 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889210 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889213 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889215 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889218 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889220 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889223 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889225 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889228 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889230 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889233 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889235 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889238 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889241 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889249 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889252 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889254 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 13:57:48.892326 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889257 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 13:57:48.892790 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.889262 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 06 13:57:48.892790 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889409 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 13:57:48.892790 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889414 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 13:57:48.892790 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889417 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 13:57:48.892790 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889420 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 13:57:48.892790 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889423 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 13:57:48.892790 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889425 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 13:57:48.892790 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889428 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 13:57:48.892790 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889431 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 13:57:48.892790 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889433 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 13:57:48.892790 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889436 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 13:57:48.892790 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889439 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 13:57:48.892790 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889441 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 13:57:48.892790 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889444 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889448 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889451 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889455 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889458 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889461 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889464 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889467 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889469 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889472 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889475 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889478 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889481 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889483 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889487 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889490 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889498 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889501 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 13:57:48.893263 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889504 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889506 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889509 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889511 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889514 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889516 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889519 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889522 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889524 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889527 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889529 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889532 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889534 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889537 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889539 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889542 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889545 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889547 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889550 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889552 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 13:57:48.893757 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889554 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889557 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889561 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889564 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889567 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889570 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889573 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889577 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889580 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889583 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889585 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889596 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889599 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889601 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889603 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889606 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889608 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889611 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889614 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889616 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 13:57:48.894481 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889619 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889621 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889624 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889632 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889635 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889637 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889640 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889643 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889645 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889648 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889653 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889656 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889658 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889661 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889664 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889666 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889668 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889671 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889674 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889676 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889679 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 13:57:48.895227 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889681 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 13:57:48.895789 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889684 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 13:57:48.895789 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889687 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 13:57:48.895789 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889696 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 13:57:48.895789 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889700 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 13:57:48.895789 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889703 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 13:57:48.895789 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889705 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 13:57:48.895789 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:48.889708 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 13:57:48.895789 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.889713 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 06 13:57:48.895789 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.889832 2577 server.go:962] "Client rotation is on, will bootstrap in background" Sep 06 13:57:48.895789 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.892047 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 06 13:57:48.895789 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.892917 2577 server.go:1019] "Starting client certificate rotation" Sep 06 13:57:48.895789 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.893045 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 06 13:57:48.895789 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.893097 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 06 13:57:48.896184 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.895455 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 06 13:57:48.900185 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.900113 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 06 13:57:48.909981 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.909961 2577 log.go:25] "Validated CRI v1 runtime API" Sep 06 13:57:48.913093 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.913079 2577 log.go:25] "Validated CRI v1 image API" Sep 06 13:57:48.913926 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.913900 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 06 13:57:48.918641 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.918618 2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a935b858-81ee-44a4-8977-0a29e13d6e2a:/dev/nvme0n1p4 e0baeb01-c33f-47fa-a2da-a8f504837822:/dev/nvme0n1p3] Sep 06 13:57:48.918711 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.918639 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 06 13:57:48.920287 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.920269 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 06 13:57:48.923831 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.923682 2577 manager.go:216] Machine: {Timestamp:2026-09-06 13:57:48.922383937 +0000 UTC m=+0.377872790 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099466 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec262a8db38453952a17c22edad936e4 SystemUUID:ec262a8d-b384-5395-2a17-c22edad936e4 BootID:ffadcbce-c5af-4e74-bf16-34e6e89f82a4 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:da:ef:db:8a:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:da:ef:db:8a:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:fc:38:7b:b2:c5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 06 13:57:48.923831 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.923781 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 06 13:57:48.923939 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.923883 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 06 13:57:48.926593 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.926568 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 06 13:57:48.926752 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.926595 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-113.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 06 13:57:48.927440 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.927428 2577 topology_manager.go:138] "Creating topology manager with none policy" Sep 06 13:57:48.927495 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.927447 2577 container_manager_linux.go:309] "Creating device plugin manager" Sep 06 13:57:48.927495 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.927457 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 06 13:57:48.928136 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.928124 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 06 13:57:48.929490 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.929479 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 06 13:57:48.933577 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.933564 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 06 13:57:48.933710 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.933699 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 06 13:57:48.933772 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.933737 2577 kubelet.go:486] "Attempting to sync node with API server" Sep 06 13:57:48.933772 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.933752 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 06 13:57:48.933772 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.933766 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 06 13:57:48.933899 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.933823 2577 kubelet.go:398] "Adding apiserver pod source" Sep 06 13:57:48.933899 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.933837 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 06 13:57:48.934200 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.934186 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 06 13:57:48.934270 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.934210 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 06 13:57:48.936674 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.936660 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 06 13:57:48.939065 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.939051 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 06 13:57:48.939118 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.939082 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 06 13:57:48.939306 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.939292 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 06 13:57:48.939338 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.939315 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 06 13:57:48.939338 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.939325 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 06 13:57:48.939338 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.939330 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 06 13:57:48.939338 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.939338 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 06 13:57:48.939445 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.939343 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 06 13:57:48.939804 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.939795 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 06 13:57:48.939833 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.939815 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 06 13:57:48.939833 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.939821 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 06 13:57:48.939833 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.939827 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 06 13:57:48.939948 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.939844 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 06 13:57:48.939948 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.939853 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 06 13:57:48.941773 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.941759 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 06 13:57:48.941773 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.941774 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 06 13:57:48.942718 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.942702 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f752t" Sep 06 13:57:48.945639 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.945621 2577 server.go:1268] "Started kubelet" Sep 06 13:57:48.945751 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.945721 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 06 13:57:48.946163 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.945926 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 06 13:57:48.946265 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.946188 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 06 13:57:48.946499 ip-10-0-132-113 systemd[1]: Started Kubernetes Kubelet. Sep 06 13:57:48.949314 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.948537 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 06 13:57:48.949411 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.949323 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-113.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 06 13:57:48.950593 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:48.950494 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 06 13:57:48.951036 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:48.950876 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-113.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 06 13:57:48.951294 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.951273 2577 server.go:310] "Adding debug handlers to kubelet server" Sep 06 13:57:48.951420 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.951389 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f752t" Sep 06 13:57:48.952079 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:48.952050 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 06 13:57:48.956036 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:48.955107 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-113.ec2.internal.18d2bff6d927427b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-113.ec2.internal,UID:ip-10-0-132-113.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-113.ec2.internal,},FirstTimestamp:2026-09-06 13:57:48.945597051 +0000 UTC m=+0.401085904,LastTimestamp:2026-09-06 13:57:48.945597051 +0000 UTC m=+0.401085904,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-113.ec2.internal,}" Sep 06 13:57:48.957331 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.957309 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 06 13:57:48.957878 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.957854 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 06 13:57:48.957962 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:48.957945 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-113.ec2.internal\" not found" Sep 06 13:57:48.958595 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.958575 2577 factory.go:55] Registering systemd factory Sep 06 13:57:48.958688 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.958610 2577 factory.go:223] Registration of the systemd container factory successfully Sep 06 13:57:48.958688 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.958613 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 06 13:57:48.958688 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.958630 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 06 13:57:48.958688 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.958612 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 06 13:57:48.958859 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.958755 2577 reconstruct.go:97] "Volume reconstruction finished" Sep 06 13:57:48.958859 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.958765 2577 reconciler.go:29] "Reconciler: start to sync state" Sep 06 13:57:48.958859 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.958813 2577 factory.go:153] Registering CRI-O factory Sep 06 13:57:48.958859 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.958823 2577 factory.go:223] Registration of the crio container factory successfully Sep 06 13:57:48.959065 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.958869 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 06 13:57:48.959065 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.958882 2577 factory.go:103] Registering Raw factory Sep 06 13:57:48.959065 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.958891 2577 manager.go:1196] Started watching for new ooms in manager Sep 06 13:57:48.959290 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.959278 2577 manager.go:349] Starting recovery of all containers Sep 06 13:57:48.966140 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.966118 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 13:57:48.968420 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:48.968394 2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-113.ec2.internal\" not found" node="ip-10-0-132-113.ec2.internal" Sep 06 13:57:48.969463 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.969444 2577 manager.go:354] Recovery completed Sep 06 13:57:48.974532 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.973937 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 13:57:48.975773 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.975757 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasSufficientMemory" Sep 06 13:57:48.975849 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.975785 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 13:57:48.975849 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.975797 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasSufficientPID" Sep 06 13:57:48.976320 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.976308 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 06 13:57:48.976370 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.976320 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 06 13:57:48.976370 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.976335 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 06 13:57:48.976435 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.976311 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 13:57:48.977081 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.977060 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasSufficientMemory" Sep 06 13:57:48.977147 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.977090 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 13:57:48.977147 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.977105 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasSufficientPID" Sep 06 13:57:48.978389 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.978377 2577 policy_none.go:49] "None policy: Start" Sep 06 13:57:48.978440 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.978392 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 06 13:57:48.978440 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.978401 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 06 13:57:48.980233 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:48.980222 2577 policy_none.go:47] "Start" Sep 06 13:57:49.007514 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.007500 2577 manager.go:337] "Starting Device Plugin manager" Sep 06 13:57:49.007579 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.007532 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 06 13:57:49.007579 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.007541 2577 server.go:87] "Starting device plugin registration server" Sep 06 13:57:49.007789 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.007773 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 06 13:57:49.007876 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.007788 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 06 13:57:49.007876 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.007863 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 06 13:57:49.008010 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.007962 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 06 13:57:49.008010 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.007971 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 06 13:57:49.008569 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.008550 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 06 13:57:49.008633 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.008602 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-113.ec2.internal\" not found" Sep 06 13:57:49.059661 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.059629 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 06 13:57:49.061377 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.060827 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 06 13:57:49.061377 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.060873 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 06 13:57:49.061377 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.060894 2577 kubelet.go:2448] "Starting kubelet main sync loop" Sep 06 13:57:49.061377 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.060952 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 06 13:57:49.062667 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.062652 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 13:57:49.108473 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.108453 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 13:57:49.109313 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.109297 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasSufficientMemory" Sep 06 13:57:49.109391 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.109326 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 13:57:49.109391 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.109343 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasSufficientPID" Sep 06 13:57:49.109391 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.109370 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.116347 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.116331 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.116426 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.116351 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-113.ec2.internal\": node \"ip-10-0-132-113.ec2.internal\" not found" Sep 06 13:57:49.124889 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.124819 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-113.ec2.internal\" not found" Sep 06 13:57:49.161106 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.161081 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-113.ec2.internal"] Sep 06 13:57:49.161191 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.161155 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 13:57:49.162632 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.162617 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasSufficientMemory" Sep 06 13:57:49.162705 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.162648 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 13:57:49.162705 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.162658 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasSufficientPID" Sep 06 13:57:49.164635 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.164624 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 13:57:49.164750 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.164734 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.164891 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.164767 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 13:57:49.165376 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.165356 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasSufficientMemory" Sep 06 13:57:49.165450 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.165386 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 13:57:49.165450 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.165396 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasSufficientPID" Sep 06 13:57:49.165450 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.165356 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasSufficientMemory" Sep 06 13:57:49.165536 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.165455 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 13:57:49.165536 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.165468 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasSufficientPID" Sep 06 13:57:49.167351 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.167337 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.167394 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.167368 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 13:57:49.168059 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.168040 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasSufficientMemory" Sep 06 13:57:49.168149 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.168069 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 13:57:49.168149 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.168084 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeHasSufficientPID" Sep 06 13:57:49.194529 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.194508 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-113.ec2.internal\" not found" node="ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.199015 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.198999 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-113.ec2.internal\" not found" node="ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.225014 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.224996 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-113.ec2.internal\" not found" Sep 06 13:57:49.260313 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.260295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/88425535be0e3f6db9fbc0fc46a4e7fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal\" (UID: \"88425535be0e3f6db9fbc0fc46a4e7fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.260391 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.260317 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88425535be0e3f6db9fbc0fc46a4e7fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal\" (UID: \"88425535be0e3f6db9fbc0fc46a4e7fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.260391 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.260336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d4e8e4996366f70a783b63f04db7c684-config\") pod \"kube-apiserver-proxy-ip-10-0-132-113.ec2.internal\" (UID: \"d4e8e4996366f70a783b63f04db7c684\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.325836 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.325817 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-113.ec2.internal\" not found" Sep 06 13:57:49.361163 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.361143 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88425535be0e3f6db9fbc0fc46a4e7fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal\" (UID: \"88425535be0e3f6db9fbc0fc46a4e7fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.361222 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.361169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d4e8e4996366f70a783b63f04db7c684-config\") pod \"kube-apiserver-proxy-ip-10-0-132-113.ec2.internal\" (UID: \"d4e8e4996366f70a783b63f04db7c684\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.361222 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.361185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/88425535be0e3f6db9fbc0fc46a4e7fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal\" (UID: \"88425535be0e3f6db9fbc0fc46a4e7fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.361288 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.361240 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88425535be0e3f6db9fbc0fc46a4e7fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal\" (UID: \"88425535be0e3f6db9fbc0fc46a4e7fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.361288 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.361244 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/88425535be0e3f6db9fbc0fc46a4e7fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal\" (UID: \"88425535be0e3f6db9fbc0fc46a4e7fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.361288 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.361246 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d4e8e4996366f70a783b63f04db7c684-config\") pod \"kube-apiserver-proxy-ip-10-0-132-113.ec2.internal\" (UID: \"d4e8e4996366f70a783b63f04db7c684\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.426419 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.426360 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-113.ec2.internal\" not found" Sep 06 13:57:49.498029 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.498003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.502980 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.502562 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.527278 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.527244 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-113.ec2.internal\" not found" Sep 06 13:57:49.627762 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.627739 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-113.ec2.internal\" not found" Sep 06 13:57:49.728193 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.728150 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-113.ec2.internal\" not found" Sep 06 13:57:49.788429 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.788406 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 13:57:49.858884 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.858861 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.868935 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.868918 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 06 13:57:49.871034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.871019 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-113.ec2.internal" Sep 06 13:57:49.878983 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.878963 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 06 13:57:49.893458 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.893432 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 06 13:57:49.893588 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.893568 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 06 13:57:49.893626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.893614 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 06 13:57:49.893668 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.893616 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 06 13:57:49.934188 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.934019 2577 apiserver.go:52] "Watching apiserver" Sep 06 13:57:49.950138 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.950117 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 06 13:57:49.951415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.951393 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-113.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74","openshift-dns/node-resolver-42dlx","openshift-network-diagnostics/network-check-target-gkcs9","openshift-network-operator/iptables-alerter-g7hzl","openshift-ovn-kubernetes/ovnkube-node-mwnd7","kube-system/global-pull-secret-syncer-9fqd2","kube-system/konnectivity-agent-ms5fd","openshift-cluster-node-tuning-operator/tuned-p9w44","openshift-image-registry/node-ca-98p47","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal","openshift-multus/multus-additional-cni-plugins-7x44l","openshift-multus/multus-ddhdd","openshift-multus/network-metrics-daemon-bkxp5"] Sep 06 13:57:49.954734 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.954714 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-98p47" Sep 06 13:57:49.956829 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.956800 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-05 13:52:49 +0000 UTC" deadline="2028-02-19 05:38:06.846622508 +0000 UTC" Sep 06 13:57:49.956952 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.956828 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12735h40m16.889798277s" Sep 06 13:57:49.958419 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.957680 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 06 13:57:49.958419 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.957740 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 06 13:57:49.958419 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.957787 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 06 13:57:49.958419 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.958106 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 06 13:57:49.958419 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.958390 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tbflc\"" Sep 06 13:57:49.958724 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.958544 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.961160 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.961138 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-42dlx" Sep 06 13:57:49.962317 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.962299 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 06 13:57:49.962509 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.962494 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ml9b7\"" Sep 06 13:57:49.962819 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.962803 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 06 13:57:49.962867 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.962817 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 06 13:57:49.962944 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.962928 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 06 13:57:49.963244 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.963229 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:57:49.963330 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.963309 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:57:49.963470 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.963454 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 06 13:57:49.963518 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.963472 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 06 13:57:49.963659 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.963647 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cxmpn\"" Sep 06 13:57:49.964753 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.964737 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-multus-conf-dir\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.964811 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.964761 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c264c419-5ce3-43d4-b604-8580a52f3518-multus-daemon-config\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.964811 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.964777 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-run-multus-certs\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.964811 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.964797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-etc-kubernetes\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.964954 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.964823 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqcmz\" (UniqueName: \"kubernetes.io/projected/c264c419-5ce3-43d4-b604-8580a52f3518-kube-api-access-tqcmz\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.964954 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.964846 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-multus-socket-dir-parent\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.964954 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.964890 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8d2l\" (UniqueName: \"kubernetes.io/projected/82c12af7-6d5b-4f10-871c-442eea180684-kube-api-access-x8d2l\") pod \"node-ca-98p47\" (UID: \"82c12af7-6d5b-4f10-871c-442eea180684\") " pod="openshift-image-registry/node-ca-98p47" Sep 06 13:57:49.964954 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.964933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-system-cni-dir\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.964954 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.964950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-multus-cni-dir\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.965120 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.964964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c264c419-5ce3-43d4-b604-8580a52f3518-cni-binary-copy\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.965120 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.964983 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-run-k8s-cni-cncf-io\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.965120 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.965002 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-run-netns\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.965120 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.965018 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61206fdd-09e3-449e-b320-c62e226616e0-tmp-dir\") pod \"node-resolver-42dlx\" (UID: \"61206fdd-09e3-449e-b320-c62e226616e0\") " pod="openshift-dns/node-resolver-42dlx" Sep 06 13:57:49.965120 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.965033 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82c12af7-6d5b-4f10-871c-442eea180684-host\") pod \"node-ca-98p47\" (UID: \"82c12af7-6d5b-4f10-871c-442eea180684\") " pod="openshift-image-registry/node-ca-98p47" Sep 06 13:57:49.965120 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.965074 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82c12af7-6d5b-4f10-871c-442eea180684-serviceca\") pod \"node-ca-98p47\" (UID: \"82c12af7-6d5b-4f10-871c-442eea180684\") " pod="openshift-image-registry/node-ca-98p47" Sep 06 13:57:49.965120 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.965104 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-cnibin\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.965348 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.965128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-var-lib-cni-bin\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.965348 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.965149 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-var-lib-kubelet\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.965348 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.965168 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-hostroot\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.965348 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.965184 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61206fdd-09e3-449e-b320-c62e226616e0-hosts-file\") pod \"node-resolver-42dlx\" (UID: \"61206fdd-09e3-449e-b320-c62e226616e0\") " pod="openshift-dns/node-resolver-42dlx" Sep 06 13:57:49.965348 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.965206 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpx5g\" (UniqueName: \"kubernetes.io/projected/61206fdd-09e3-449e-b320-c62e226616e0-kube-api-access-tpx5g\") pod \"node-resolver-42dlx\" (UID: \"61206fdd-09e3-449e-b320-c62e226616e0\") " pod="openshift-dns/node-resolver-42dlx" Sep 06 13:57:49.965348 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.965227 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-var-lib-cni-multus\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:49.965622 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.965474 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g7hzl" Sep 06 13:57:49.967684 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.967663 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bzfm2\"" Sep 06 13:57:49.967768 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.967750 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:49.967821 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.967806 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 06 13:57:49.967888 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.967868 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 06 13:57:49.967954 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.967874 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 06 13:57:49.970139 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.970124 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 06 13:57:49.970222 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.970168 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:49.970393 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.970357 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 06 13:57:49.970523 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.970503 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w2tnm\"" Sep 06 13:57:49.970642 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.970627 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 06 13:57:49.970749 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.970715 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 06 13:57:49.970749 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.970721 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 06 13:57:49.970894 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.970755 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 06 13:57:49.972506 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.972487 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:57:49.972610 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.972556 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:57:49.972610 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.972579 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fg5mg\"" Sep 06 13:57:49.972610 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.972595 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 06 13:57:49.972769 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.972604 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 06 13:57:49.975010 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.974994 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:49.975433 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.975413 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 06 13:57:49.977377 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.977357 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ms5fd" Sep 06 13:57:49.977476 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.977430 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 06 13:57:49.977476 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.977452 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 06 13:57:49.977598 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.977449 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-drm4s\"" Sep 06 13:57:49.977598 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.977435 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 06 13:57:49.979749 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.979705 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v6h5m\"" Sep 06 13:57:49.979887 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.979867 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:49.980169 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.980062 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 06 13:57:49.980169 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.980104 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 06 13:57:49.982859 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.982830 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 06 13:57:49.982976 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.982877 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fnwx9\"" Sep 06 13:57:49.982976 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.982893 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 06 13:57:49.983868 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.983849 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:49.983953 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:49.983902 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:57:49.998832 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:49.998808 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kkbhj" Sep 06 13:57:50.006169 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.006144 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kkbhj" Sep 06 13:57:50.059722 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.059703 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 06 13:57:50.065543 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-run-k8s-cni-cncf-io\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.065632 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065547 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-run-netns\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.065632 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065568 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-lib-modules\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.065632 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065584 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-run-systemd\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.065632 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065607 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-sysctl-d\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.065840 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61206fdd-09e3-449e-b320-c62e226616e0-hosts-file\") pod \"node-resolver-42dlx\" (UID: \"61206fdd-09e3-449e-b320-c62e226616e0\") " pod="openshift-dns/node-resolver-42dlx" Sep 06 13:57:50.065840 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065674 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-run-k8s-cni-cncf-io\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.065840 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065707 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpx5g\" (UniqueName: \"kubernetes.io/projected/61206fdd-09e3-449e-b320-c62e226616e0-kube-api-access-tpx5g\") pod \"node-resolver-42dlx\" (UID: \"61206fdd-09e3-449e-b320-c62e226616e0\") " pod="openshift-dns/node-resolver-42dlx" Sep 06 13:57:50.065840 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065710 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-run-netns\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.065840 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065738 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61206fdd-09e3-449e-b320-c62e226616e0-hosts-file\") pod \"node-resolver-42dlx\" (UID: \"61206fdd-09e3-449e-b320-c62e226616e0\") " pod="openshift-dns/node-resolver-42dlx" Sep 06 13:57:50.065840 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-systemd-units\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.065840 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-log-socket\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.065840 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-kubernetes\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.065840 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065837 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-run\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-var-lib-cni-multus\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-etc-selinux\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-systemd\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065953 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-multus-conf-dir\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-system-cni-dir\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065988 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-etc-openvswitch\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065989 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-multus-conf-dir\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.065951 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-var-lib-cni-multus\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066011 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f579\" (UniqueName: \"kubernetes.io/projected/c6753921-b1e1-433c-8065-f10c62e274f4-kube-api-access-9f579\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-device-dir\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066062 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-system-cni-dir\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066070 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdb2dbef-18c2-4a14-a421-f15c0286c411-system-cni-dir\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdb2dbef-18c2-4a14-a421-f15c0286c411-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066110 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61206fdd-09e3-449e-b320-c62e226616e0-tmp-dir\") pod \"node-resolver-42dlx\" (UID: \"61206fdd-09e3-449e-b320-c62e226616e0\") " pod="openshift-dns/node-resolver-42dlx" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82c12af7-6d5b-4f10-871c-442eea180684-serviceca\") pod \"node-ca-98p47\" (UID: \"82c12af7-6d5b-4f10-871c-442eea180684\") " pod="openshift-image-registry/node-ca-98p47" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-cnibin\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.066210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066164 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-run-openvswitch\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066188 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6753921-b1e1-433c-8065-f10c62e274f4-ovn-node-metrics-cert\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066215 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-sys-fs\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066233 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-cnibin\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066249 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23a987fd-669e-4094-9025-a403d8be2a2d-host-slash\") pod \"iptables-alerter-g7hzl\" (UID: \"23a987fd-669e-4094-9025-a403d8be2a2d\") " pod="openshift-network-operator/iptables-alerter-g7hzl" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-modprobe-d\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab2f34a9-090c-4b27-a511-310e6d0e50ed-konnectivity-ca\") pod \"konnectivity-agent-ms5fd\" (UID: \"ab2f34a9-090c-4b27-a511-310e6d0e50ed\") " pod="kube-system/konnectivity-agent-ms5fd" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-var-lib-kubelet\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68e4ea42-b401-4f77-844d-1eee77f1d8b0-tmp\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066381 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cdb2dbef-18c2-4a14-a421-f15c0286c411-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066409 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cdb2dbef-18c2-4a14-a421-f15c0286c411-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066419 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61206fdd-09e3-449e-b320-c62e226616e0-tmp-dir\") pod \"node-resolver-42dlx\" (UID: \"61206fdd-09e3-449e-b320-c62e226616e0\") " pod="openshift-dns/node-resolver-42dlx" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066437 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl5wg\" (UniqueName: \"kubernetes.io/projected/cdb2dbef-18c2-4a14-a421-f15c0286c411-kube-api-access-nl5wg\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/53540329-a02b-4fcb-ac80-8d6bf3489b10-kubelet-config\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066485 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9rf\" (UniqueName: \"kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf\") pod \"network-check-target-gkcs9\" (UID: \"83eae010-1fa5-4861-ad26-9be01d46a741\") " pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066502 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-etc-kubernetes\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.066910 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066527 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-slash\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066535 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-etc-kubernetes\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066540 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82c12af7-6d5b-4f10-871c-442eea180684-serviceca\") pod \"node-ca-98p47\" (UID: \"82c12af7-6d5b-4f10-871c-442eea180684\") " pod="openshift-image-registry/node-ca-98p47" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066544 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-registration-dir\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066582 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-run-multus-certs\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066608 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-kubelet\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-multus-socket-dir-parent\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066626 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-run-multus-certs\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-multus-cni-dir\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066663 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c264c419-5ce3-43d4-b604-8580a52f3518-cni-binary-copy\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066677 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-hostroot\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066700 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6753921-b1e1-433c-8065-f10c62e274f4-ovnkube-config\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066717 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-sysconfig\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066719 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-multus-socket-dir-parent\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82c12af7-6d5b-4f10-871c-442eea180684-host\") pod \"node-ca-98p47\" (UID: \"82c12af7-6d5b-4f10-871c-442eea180684\") " pod="openshift-image-registry/node-ca-98p47" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066756 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-hostroot\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-node-log\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066792 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.067739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab2f34a9-090c-4b27-a511-310e6d0e50ed-agent-certs\") pod \"konnectivity-agent-ms5fd\" (UID: \"ab2f34a9-090c-4b27-a511-310e6d0e50ed\") " pod="kube-system/konnectivity-agent-ms5fd" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066815 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82c12af7-6d5b-4f10-871c-442eea180684-host\") pod \"node-ca-98p47\" (UID: \"82c12af7-6d5b-4f10-871c-442eea180684\") " pod="openshift-image-registry/node-ca-98p47" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-sysctl-conf\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066850 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-multus-cni-dir\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066854 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs\") pod \"network-metrics-daemon-bkxp5\" (UID: \"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd\") " pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdb2dbef-18c2-4a14-a421-f15c0286c411-cnibin\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.066974 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-sys\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067002 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-host\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067028 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8d2l\" (UniqueName: \"kubernetes.io/projected/82c12af7-6d5b-4f10-871c-442eea180684-kube-api-access-x8d2l\") pod \"node-ca-98p47\" (UID: \"82c12af7-6d5b-4f10-871c-442eea180684\") " pod="openshift-image-registry/node-ca-98p47" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-var-lib-cni-bin\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067112 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-var-lib-openvswitch\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067137 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067161 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6753921-b1e1-433c-8065-f10c62e274f4-ovnkube-script-lib\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067191 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-var-lib-cni-bin\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067196 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj5l7\" (UniqueName: \"kubernetes.io/projected/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-kube-api-access-lj5l7\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lvr5\" (UniqueName: \"kubernetes.io/projected/23a987fd-669e-4094-9025-a403d8be2a2d-kube-api-access-9lvr5\") pod \"iptables-alerter-g7hzl\" (UID: \"23a987fd-669e-4094-9025-a403d8be2a2d\") " pod="openshift-network-operator/iptables-alerter-g7hzl" Sep 06 13:57:50.068415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067248 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c264c419-5ce3-43d4-b604-8580a52f3518-cni-binary-copy\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067269 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdb2dbef-18c2-4a14-a421-f15c0286c411-cni-binary-copy\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-var-lib-kubelet\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067341 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-run-netns\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067366 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-cni-bin\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067371 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c264c419-5ce3-43d4-b604-8580a52f3518-host-var-lib-kubelet\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/23a987fd-669e-4094-9025-a403d8be2a2d-iptables-alerter-script\") pod \"iptables-alerter-g7hzl\" (UID: \"23a987fd-669e-4094-9025-a403d8be2a2d\") " pod="openshift-network-operator/iptables-alerter-g7hzl" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067443 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-run-ovn\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/53540329-a02b-4fcb-ac80-8d6bf3489b10-dbus\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067501 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-cni-netd\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067532 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6753921-b1e1-433c-8065-f10c62e274f4-env-overrides\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-socket-dir\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067583 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-tuned\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067610 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht6th\" (UniqueName: \"kubernetes.io/projected/68e4ea42-b401-4f77-844d-1eee77f1d8b0-kube-api-access-ht6th\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bld2v\" (UniqueName: \"kubernetes.io/projected/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-kube-api-access-bld2v\") pod \"network-metrics-daemon-bkxp5\" (UID: \"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd\") " pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:57:50.068934 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c264c419-5ce3-43d4-b604-8580a52f3518-multus-daemon-config\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.069432 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.067691 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqcmz\" (UniqueName: \"kubernetes.io/projected/c264c419-5ce3-43d4-b604-8580a52f3518-kube-api-access-tqcmz\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.069432 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.068150 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c264c419-5ce3-43d4-b604-8580a52f3518-multus-daemon-config\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.074026 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.074004 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 06 13:57:50.077747 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.077729 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8d2l\" (UniqueName: \"kubernetes.io/projected/82c12af7-6d5b-4f10-871c-442eea180684-kube-api-access-x8d2l\") pod \"node-ca-98p47\" (UID: \"82c12af7-6d5b-4f10-871c-442eea180684\") " pod="openshift-image-registry/node-ca-98p47" Sep 06 13:57:50.077846 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.077774 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpx5g\" (UniqueName: \"kubernetes.io/projected/61206fdd-09e3-449e-b320-c62e226616e0-kube-api-access-tpx5g\") pod \"node-resolver-42dlx\" (UID: \"61206fdd-09e3-449e-b320-c62e226616e0\") " pod="openshift-dns/node-resolver-42dlx" Sep 06 13:57:50.077894 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.077848 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqcmz\" (UniqueName: \"kubernetes.io/projected/c264c419-5ce3-43d4-b604-8580a52f3518-kube-api-access-tqcmz\") pod \"multus-ddhdd\" (UID: \"c264c419-5ce3-43d4-b604-8580a52f3518\") " pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.129863 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:50.129838 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e8e4996366f70a783b63f04db7c684.slice/crio-5300eac430cb943396970e949244ade4b5be9acbf8534ca2030bb5c0d6acc737 WatchSource:0}: Error finding container 5300eac430cb943396970e949244ade4b5be9acbf8534ca2030bb5c0d6acc737: Status 404 returned error can't find the container with id 5300eac430cb943396970e949244ade4b5be9acbf8534ca2030bb5c0d6acc737 Sep 06 13:57:50.130103 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:50.130080 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88425535be0e3f6db9fbc0fc46a4e7fd.slice/crio-50a063d9394d31407aa4aa0754d758ba0c7d6ffbefa254f41190c8a853f636a2 WatchSource:0}: Error finding container 50a063d9394d31407aa4aa0754d758ba0c7d6ffbefa254f41190c8a853f636a2: Status 404 returned error can't find the container with id 50a063d9394d31407aa4aa0754d758ba0c7d6ffbefa254f41190c8a853f636a2 Sep 06 13:57:50.136076 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.136060 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 13:57:50.148172 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.148151 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 13:57:50.168668 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-registration-dir\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.168771 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-kubelet\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.168771 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168701 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6753921-b1e1-433c-8065-f10c62e274f4-ovnkube-config\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.168771 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-sysconfig\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.168771 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-node-log\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.168771 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168760 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-kubelet\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168762 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-registration-dir\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168795 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-sysconfig\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab2f34a9-090c-4b27-a511-310e6d0e50ed-agent-certs\") pod \"konnectivity-agent-ms5fd\" (UID: \"ab2f34a9-090c-4b27-a511-310e6d0e50ed\") " pod="kube-system/konnectivity-agent-ms5fd" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168824 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-node-log\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168852 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-sysctl-conf\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168859 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs\") pod \"network-metrics-daemon-bkxp5\" (UID: \"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd\") " pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdb2dbef-18c2-4a14-a421-f15c0286c411-cnibin\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168943 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-sys\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-host\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168968 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdb2dbef-18c2-4a14-a421-f15c0286c411-cnibin\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168977 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-var-lib-openvswitch\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169011 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-sys\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.168942 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-sysctl-conf\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.169064 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169028 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169040 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-host\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.169092 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6753921-b1e1-433c-8065-f10c62e274f4-ovnkube-script-lib\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.169139 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169149 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.169197 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret podName:53540329-a02b-4fcb-ac80-8d6bf3489b10 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:50.669152811 +0000 UTC m=+2.124641681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret") pod "global-pull-secret-syncer-9fqd2" (UID: "53540329-a02b-4fcb-ac80-8d6bf3489b10") : object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169213 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-var-lib-openvswitch\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169224 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj5l7\" (UniqueName: \"kubernetes.io/projected/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-kube-api-access-lj5l7\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169258 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6753921-b1e1-433c-8065-f10c62e274f4-ovnkube-config\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.169280 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs podName:a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd nodeName:}" failed. No retries permitted until 2026-09-06 13:57:50.669267746 +0000 UTC m=+2.124756591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs") pod "network-metrics-daemon-bkxp5" (UID: "a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169300 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvr5\" (UniqueName: \"kubernetes.io/projected/23a987fd-669e-4094-9025-a403d8be2a2d-kube-api-access-9lvr5\") pod \"iptables-alerter-g7hzl\" (UID: \"23a987fd-669e-4094-9025-a403d8be2a2d\") " pod="openshift-network-operator/iptables-alerter-g7hzl" Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdb2dbef-18c2-4a14-a421-f15c0286c411-cni-binary-copy\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-run-netns\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169467 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-cni-bin\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169502 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-run-netns\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.169908 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169515 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-cni-bin\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169550 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/23a987fd-669e-4094-9025-a403d8be2a2d-iptables-alerter-script\") pod \"iptables-alerter-g7hzl\" (UID: \"23a987fd-669e-4094-9025-a403d8be2a2d\") " pod="openshift-network-operator/iptables-alerter-g7hzl" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169603 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-run-ovn\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/53540329-a02b-4fcb-ac80-8d6bf3489b10-dbus\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-cni-netd\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169671 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-run-ovn\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169680 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6753921-b1e1-433c-8065-f10c62e274f4-env-overrides\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-socket-dir\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169711 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-cni-netd\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169716 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6753921-b1e1-433c-8065-f10c62e274f4-ovnkube-script-lib\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169716 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/53540329-a02b-4fcb-ac80-8d6bf3489b10-dbus\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169797 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-socket-dir\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169822 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-tuned\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169846 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht6th\" (UniqueName: \"kubernetes.io/projected/68e4ea42-b401-4f77-844d-1eee77f1d8b0-kube-api-access-ht6th\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.169874 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bld2v\" (UniqueName: \"kubernetes.io/projected/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-kube-api-access-bld2v\") pod \"network-metrics-daemon-bkxp5\" (UID: \"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd\") " pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:57:50.170742 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170008 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdb2dbef-18c2-4a14-a421-f15c0286c411-cni-binary-copy\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170011 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-lib-modules\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170062 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-run-systemd\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-sysctl-d\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170096 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-lib-modules\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170087 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6753921-b1e1-433c-8065-f10c62e274f4-env-overrides\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-systemd-units\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170150 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-run-systemd\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170156 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-systemd-units\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-log-socket\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170191 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-kubernetes\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170204 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-sysctl-d\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-run\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170241 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-run\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-etc-selinux\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-systemd\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170289 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-kubernetes\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170310 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-etc-openvswitch\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.171546 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170314 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-etc-selinux\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170253 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-log-socket\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f579\" (UniqueName: \"kubernetes.io/projected/c6753921-b1e1-433c-8065-f10c62e274f4-kube-api-access-9f579\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170347 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-systemd\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-device-dir\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170386 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-device-dir\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170359 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-etc-openvswitch\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdb2dbef-18c2-4a14-a421-f15c0286c411-system-cni-dir\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170434 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdb2dbef-18c2-4a14-a421-f15c0286c411-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170437 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdb2dbef-18c2-4a14-a421-f15c0286c411-system-cni-dir\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-run-openvswitch\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6753921-b1e1-433c-8065-f10c62e274f4-ovn-node-metrics-cert\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-sys-fs\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170526 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23a987fd-669e-4094-9025-a403d8be2a2d-host-slash\") pod \"iptables-alerter-g7hzl\" (UID: \"23a987fd-669e-4094-9025-a403d8be2a2d\") " pod="openshift-network-operator/iptables-alerter-g7hzl" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170551 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-modprobe-d\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170555 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-run-openvswitch\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170575 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab2f34a9-090c-4b27-a511-310e6d0e50ed-konnectivity-ca\") pod \"konnectivity-agent-ms5fd\" (UID: \"ab2f34a9-090c-4b27-a511-310e6d0e50ed\") " pod="kube-system/konnectivity-agent-ms5fd" Sep 06 13:57:50.172145 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-var-lib-kubelet\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170631 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-sys-fs\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170652 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68e4ea42-b401-4f77-844d-1eee77f1d8b0-tmp\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cdb2dbef-18c2-4a14-a421-f15c0286c411-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170686 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-modprobe-d\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170671 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23a987fd-669e-4094-9025-a403d8be2a2d-host-slash\") pod \"iptables-alerter-g7hzl\" (UID: \"23a987fd-669e-4094-9025-a403d8be2a2d\") " pod="openshift-network-operator/iptables-alerter-g7hzl" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cdb2dbef-18c2-4a14-a421-f15c0286c411-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl5wg\" (UniqueName: \"kubernetes.io/projected/cdb2dbef-18c2-4a14-a421-f15c0286c411-kube-api-access-nl5wg\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/53540329-a02b-4fcb-ac80-8d6bf3489b10-kubelet-config\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9rf\" (UniqueName: \"kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf\") pod \"network-check-target-gkcs9\" (UID: \"83eae010-1fa5-4861-ad26-9be01d46a741\") " pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-slash\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170910 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdb2dbef-18c2-4a14-a421-f15c0286c411-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170940 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6753921-b1e1-433c-8065-f10c62e274f4-host-slash\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.170957 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/23a987fd-669e-4094-9025-a403d8be2a2d-iptables-alerter-script\") pod \"iptables-alerter-g7hzl\" (UID: \"23a987fd-669e-4094-9025-a403d8be2a2d\") " pod="openshift-network-operator/iptables-alerter-g7hzl" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.171119 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/53540329-a02b-4fcb-ac80-8d6bf3489b10-kubelet-config\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.171139 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68e4ea42-b401-4f77-844d-1eee77f1d8b0-var-lib-kubelet\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.171260 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab2f34a9-090c-4b27-a511-310e6d0e50ed-konnectivity-ca\") pod \"konnectivity-agent-ms5fd\" (UID: \"ab2f34a9-090c-4b27-a511-310e6d0e50ed\") " pod="kube-system/konnectivity-agent-ms5fd" Sep 06 13:57:50.172690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.171637 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cdb2dbef-18c2-4a14-a421-f15c0286c411-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.173247 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.171714 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cdb2dbef-18c2-4a14-a421-f15c0286c411-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.173247 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.171717 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab2f34a9-090c-4b27-a511-310e6d0e50ed-agent-certs\") pod \"konnectivity-agent-ms5fd\" (UID: \"ab2f34a9-090c-4b27-a511-310e6d0e50ed\") " pod="kube-system/konnectivity-agent-ms5fd" Sep 06 13:57:50.173247 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.172421 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/68e4ea42-b401-4f77-844d-1eee77f1d8b0-etc-tuned\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.173247 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.172945 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68e4ea42-b401-4f77-844d-1eee77f1d8b0-tmp\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.173247 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.173053 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6753921-b1e1-433c-8065-f10c62e274f4-ovn-node-metrics-cert\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.178778 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.178760 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 13:57:50.178778 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.178780 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 13:57:50.178942 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.178789 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xq9rf for pod openshift-network-diagnostics/network-check-target-gkcs9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:50.178942 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.178878 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf podName:83eae010-1fa5-4861-ad26-9be01d46a741 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:50.678862815 +0000 UTC m=+2.134351680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xq9rf" (UniqueName: "kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf") pod "network-check-target-gkcs9" (UID: "83eae010-1fa5-4861-ad26-9be01d46a741") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:50.179969 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.179944 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lvr5\" (UniqueName: \"kubernetes.io/projected/23a987fd-669e-4094-9025-a403d8be2a2d-kube-api-access-9lvr5\") pod \"iptables-alerter-g7hzl\" (UID: \"23a987fd-669e-4094-9025-a403d8be2a2d\") " pod="openshift-network-operator/iptables-alerter-g7hzl" Sep 06 13:57:50.180170 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.180144 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj5l7\" (UniqueName: \"kubernetes.io/projected/ff9cd7ae-4a5e-45ef-9cf5-52edb8424978-kube-api-access-lj5l7\") pod \"aws-ebs-csi-driver-node-hdz74\" (UID: \"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.180675 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.180657 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht6th\" (UniqueName: \"kubernetes.io/projected/68e4ea42-b401-4f77-844d-1eee77f1d8b0-kube-api-access-ht6th\") pod \"tuned-p9w44\" (UID: \"68e4ea42-b401-4f77-844d-1eee77f1d8b0\") " pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.180999 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.180970 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f579\" (UniqueName: \"kubernetes.io/projected/c6753921-b1e1-433c-8065-f10c62e274f4-kube-api-access-9f579\") pod \"ovnkube-node-mwnd7\" (UID: \"c6753921-b1e1-433c-8065-f10c62e274f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.181685 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.181665 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl5wg\" (UniqueName: \"kubernetes.io/projected/cdb2dbef-18c2-4a14-a421-f15c0286c411-kube-api-access-nl5wg\") pod \"multus-additional-cni-plugins-7x44l\" (UID: \"cdb2dbef-18c2-4a14-a421-f15c0286c411\") " pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.181770 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.181718 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bld2v\" (UniqueName: \"kubernetes.io/projected/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-kube-api-access-bld2v\") pod \"network-metrics-daemon-bkxp5\" (UID: \"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd\") " pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:57:50.284764 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.284701 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-98p47" Sep 06 13:57:50.292072 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.292050 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ddhdd" Sep 06 13:57:50.295040 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:50.294950 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c12af7_6d5b_4f10_871c_442eea180684.slice/crio-878032f52b95ed0afa552e1aeda11d26a12135061031409d21c50c426e66e1d3 WatchSource:0}: Error finding container 878032f52b95ed0afa552e1aeda11d26a12135061031409d21c50c426e66e1d3: Status 404 returned error can't find the container with id 878032f52b95ed0afa552e1aeda11d26a12135061031409d21c50c426e66e1d3 Sep 06 13:57:50.297481 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.297466 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-42dlx" Sep 06 13:57:50.300689 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:50.300672 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc264c419_5ce3_43d4_b604_8580a52f3518.slice/crio-efd16544e2a10c93f0541b908aa45f8d36c69f22cf97ddde280b96e9908e1d6a WatchSource:0}: Error finding container efd16544e2a10c93f0541b908aa45f8d36c69f22cf97ddde280b96e9908e1d6a: Status 404 returned error can't find the container with id efd16544e2a10c93f0541b908aa45f8d36c69f22cf97ddde280b96e9908e1d6a Sep 06 13:57:50.306004 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:50.305950 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61206fdd_09e3_449e_b320_c62e226616e0.slice/crio-332691bdeabc40c035a4fedfd8267455d8a57fcf33066cbcadfde9846f75c974 WatchSource:0}: Error finding container 332691bdeabc40c035a4fedfd8267455d8a57fcf33066cbcadfde9846f75c974: Status 404 returned error can't find the container with id 332691bdeabc40c035a4fedfd8267455d8a57fcf33066cbcadfde9846f75c974 Sep 06 13:57:50.321696 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.321677 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g7hzl" Sep 06 13:57:50.325420 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.325403 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:57:50.331676 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:50.331657 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a987fd_669e_4094_9025_a403d8be2a2d.slice/crio-8fd02be91852a933a6cd446f1315f3ec0614ee80015cc402749581335f3835b5 WatchSource:0}: Error finding container 8fd02be91852a933a6cd446f1315f3ec0614ee80015cc402749581335f3835b5: Status 404 returned error can't find the container with id 8fd02be91852a933a6cd446f1315f3ec0614ee80015cc402749581335f3835b5 Sep 06 13:57:50.334842 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:50.334823 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6753921_b1e1_433c_8065_f10c62e274f4.slice/crio-edbec74e52e035720bab7501757e6cbc94b3f82ad02fa9637098a8d246a593d4 WatchSource:0}: Error finding container edbec74e52e035720bab7501757e6cbc94b3f82ad02fa9637098a8d246a593d4: Status 404 returned error can't find the container with id edbec74e52e035720bab7501757e6cbc94b3f82ad02fa9637098a8d246a593d4 Sep 06 13:57:50.353921 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.353888 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p9w44" Sep 06 13:57:50.362490 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:50.362471 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e4ea42_b401_4f77_844d_1eee77f1d8b0.slice/crio-1111a6c719c6c313d4345764dc60d0f259c63edceadc8b3ec360e355b95db342 WatchSource:0}: Error finding container 1111a6c719c6c313d4345764dc60d0f259c63edceadc8b3ec360e355b95db342: Status 404 returned error can't find the container with id 1111a6c719c6c313d4345764dc60d0f259c63edceadc8b3ec360e355b95db342 Sep 06 13:57:50.370518 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.370501 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" Sep 06 13:57:50.376579 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.376563 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ms5fd" Sep 06 13:57:50.378569 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:50.378551 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9cd7ae_4a5e_45ef_9cf5_52edb8424978.slice/crio-0198e583782b528f264bcf1e93f79a1d593e82c0baea3eccc0f5540e61a61b3a WatchSource:0}: Error finding container 0198e583782b528f264bcf1e93f79a1d593e82c0baea3eccc0f5540e61a61b3a: Status 404 returned error can't find the container with id 0198e583782b528f264bcf1e93f79a1d593e82c0baea3eccc0f5540e61a61b3a Sep 06 13:57:50.385148 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:50.385127 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab2f34a9_090c_4b27_a511_310e6d0e50ed.slice/crio-4e1a5229a6e498de105307f7495947466a227f87c2efa5886d09f8f5cc7cd32f WatchSource:0}: Error finding container 4e1a5229a6e498de105307f7495947466a227f87c2efa5886d09f8f5cc7cd32f: Status 404 returned error can't find the container with id 4e1a5229a6e498de105307f7495947466a227f87c2efa5886d09f8f5cc7cd32f Sep 06 13:57:50.397086 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.397064 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7x44l" Sep 06 13:57:50.405948 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:57:50.405926 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb2dbef_18c2_4a14_a421_f15c0286c411.slice/crio-67a0d93619c4cd1f6d617113f186b202ab2cab02711584f2c12eca40f185af7f WatchSource:0}: Error finding container 67a0d93619c4cd1f6d617113f186b202ab2cab02711584f2c12eca40f185af7f: Status 404 returned error can't find the container with id 67a0d93619c4cd1f6d617113f186b202ab2cab02711584f2c12eca40f185af7f Sep 06 13:57:50.675461 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.675352 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs\") pod \"network-metrics-daemon-bkxp5\" (UID: \"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd\") " pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:57:50.675461 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.675405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:50.675669 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.675530 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:50.675669 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.675532 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:50.675669 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.675603 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret podName:53540329-a02b-4fcb-ac80-8d6bf3489b10 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:51.675583803 +0000 UTC m=+3.131072661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret") pod "global-pull-secret-syncer-9fqd2" (UID: "53540329-a02b-4fcb-ac80-8d6bf3489b10") : object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:50.675669 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.675622 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs podName:a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd nodeName:}" failed. No retries permitted until 2026-09-06 13:57:51.675612421 +0000 UTC m=+3.131101263 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs") pod "network-metrics-daemon-bkxp5" (UID: "a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:50.776802 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:50.776767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9rf\" (UniqueName: \"kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf\") pod \"network-check-target-gkcs9\" (UID: \"83eae010-1fa5-4861-ad26-9be01d46a741\") " pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:57:50.777014 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.776996 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 13:57:50.777152 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.777023 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 13:57:50.777152 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.777038 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xq9rf for pod openshift-network-diagnostics/network-check-target-gkcs9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:50.777152 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:50.777098 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf podName:83eae010-1fa5-4861-ad26-9be01d46a741 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:51.777078539 +0000 UTC m=+3.232567380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xq9rf" (UniqueName: "kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf") pod "network-check-target-gkcs9" (UID: "83eae010-1fa5-4861-ad26-9be01d46a741") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:51.010273 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.010114 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 13:52:50 +0000 UTC" deadline="2028-03-14 19:28:07.203186015 +0000 UTC" Sep 06 13:57:51.010273 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.010177 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13325h30m16.193013346s" Sep 06 13:57:51.062803 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.062054 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:57:51.062803 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:51.062204 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:57:51.062803 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.062660 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:57:51.062803 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:51.062755 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:57:51.118843 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.118752 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ms5fd" event={"ID":"ab2f34a9-090c-4b27-a511-310e6d0e50ed","Type":"ContainerStarted","Data":"4e1a5229a6e498de105307f7495947466a227f87c2efa5886d09f8f5cc7cd32f"} Sep 06 13:57:51.132199 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.132148 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" event={"ID":"c6753921-b1e1-433c-8065-f10c62e274f4","Type":"ContainerStarted","Data":"edbec74e52e035720bab7501757e6cbc94b3f82ad02fa9637098a8d246a593d4"} Sep 06 13:57:51.165694 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.165656 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-42dlx" event={"ID":"61206fdd-09e3-449e-b320-c62e226616e0","Type":"ContainerStarted","Data":"332691bdeabc40c035a4fedfd8267455d8a57fcf33066cbcadfde9846f75c974"} Sep 06 13:57:51.184646 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.184610 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ddhdd" event={"ID":"c264c419-5ce3-43d4-b604-8580a52f3518","Type":"ContainerStarted","Data":"efd16544e2a10c93f0541b908aa45f8d36c69f22cf97ddde280b96e9908e1d6a"} Sep 06 13:57:51.194450 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.194419 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-98p47" event={"ID":"82c12af7-6d5b-4f10-871c-442eea180684","Type":"ContainerStarted","Data":"878032f52b95ed0afa552e1aeda11d26a12135061031409d21c50c426e66e1d3"} Sep 06 13:57:51.199564 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.199523 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-113.ec2.internal" event={"ID":"d4e8e4996366f70a783b63f04db7c684","Type":"ContainerStarted","Data":"5300eac430cb943396970e949244ade4b5be9acbf8534ca2030bb5c0d6acc737"} Sep 06 13:57:51.205038 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.205010 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7x44l" event={"ID":"cdb2dbef-18c2-4a14-a421-f15c0286c411","Type":"ContainerStarted","Data":"67a0d93619c4cd1f6d617113f186b202ab2cab02711584f2c12eca40f185af7f"} Sep 06 13:57:51.209483 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.209457 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 13:57:51.245044 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.244935 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" event={"ID":"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978","Type":"ContainerStarted","Data":"0198e583782b528f264bcf1e93f79a1d593e82c0baea3eccc0f5540e61a61b3a"} Sep 06 13:57:51.274690 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.274621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p9w44" event={"ID":"68e4ea42-b401-4f77-844d-1eee77f1d8b0","Type":"ContainerStarted","Data":"1111a6c719c6c313d4345764dc60d0f259c63edceadc8b3ec360e355b95db342"} Sep 06 13:57:51.292817 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.292787 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g7hzl" event={"ID":"23a987fd-669e-4094-9025-a403d8be2a2d","Type":"ContainerStarted","Data":"8fd02be91852a933a6cd446f1315f3ec0614ee80015cc402749581335f3835b5"} Sep 06 13:57:51.322515 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.322487 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal" event={"ID":"88425535be0e3f6db9fbc0fc46a4e7fd","Type":"ContainerStarted","Data":"50a063d9394d31407aa4aa0754d758ba0c7d6ffbefa254f41190c8a853f636a2"} Sep 06 13:57:51.399139 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.399110 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 13:57:51.684374 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.684288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs\") pod \"network-metrics-daemon-bkxp5\" (UID: \"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd\") " pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:57:51.684374 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.684338 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:51.684609 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:51.684472 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:51.684609 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:51.684473 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:51.684609 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:51.684535 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret podName:53540329-a02b-4fcb-ac80-8d6bf3489b10 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:53.684515202 +0000 UTC m=+5.140004045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret") pod "global-pull-secret-syncer-9fqd2" (UID: "53540329-a02b-4fcb-ac80-8d6bf3489b10") : object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:51.684609 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:51.684556 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs podName:a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd nodeName:}" failed. No retries permitted until 2026-09-06 13:57:53.684544079 +0000 UTC m=+5.140032924 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs") pod "network-metrics-daemon-bkxp5" (UID: "a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:51.785757 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:51.785625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9rf\" (UniqueName: \"kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf\") pod \"network-check-target-gkcs9\" (UID: \"83eae010-1fa5-4861-ad26-9be01d46a741\") " pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:57:51.785947 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:51.785833 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 13:57:51.785947 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:51.785854 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 13:57:51.785947 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:51.785866 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xq9rf for pod openshift-network-diagnostics/network-check-target-gkcs9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:51.785947 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:51.785926 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf podName:83eae010-1fa5-4861-ad26-9be01d46a741 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:53.78590703 +0000 UTC m=+5.241395888 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xq9rf" (UniqueName: "kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf") pod "network-check-target-gkcs9" (UID: "83eae010-1fa5-4861-ad26-9be01d46a741") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:52.012732 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:52.012607 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 13:52:50 +0000 UTC" deadline="2028-04-28 05:30:07.561139463 +0000 UTC" Sep 06 13:57:52.012732 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:52.012658 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14391h32m15.548486491s" Sep 06 13:57:52.061948 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:52.061387 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:52.061948 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:52.061539 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:57:52.889128 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:52.889085 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 13:57:53.064868 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:53.064836 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:57:53.065489 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:53.065003 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:57:53.065489 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:53.065404 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:57:53.065600 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:53.065501 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:57:53.706157 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:53.705245 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs\") pod \"network-metrics-daemon-bkxp5\" (UID: \"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd\") " pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:57:53.706157 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:53.705295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:53.706157 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:53.705441 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:53.706157 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:53.705501 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret podName:53540329-a02b-4fcb-ac80-8d6bf3489b10 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:57.705482069 +0000 UTC m=+9.160970926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret") pod "global-pull-secret-syncer-9fqd2" (UID: "53540329-a02b-4fcb-ac80-8d6bf3489b10") : object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:53.706157 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:53.706054 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:53.706157 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:53.706125 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs podName:a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd nodeName:}" failed. No retries permitted until 2026-09-06 13:57:57.706103634 +0000 UTC m=+9.161592494 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs") pod "network-metrics-daemon-bkxp5" (UID: "a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:53.806584 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:53.806493 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9rf\" (UniqueName: \"kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf\") pod \"network-check-target-gkcs9\" (UID: \"83eae010-1fa5-4861-ad26-9be01d46a741\") " pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:57:53.806794 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:53.806724 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 13:57:53.806794 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:53.806746 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 13:57:53.806794 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:53.806760 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xq9rf for pod openshift-network-diagnostics/network-check-target-gkcs9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:53.807065 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:53.806815 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf podName:83eae010-1fa5-4861-ad26-9be01d46a741 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:57.806796803 +0000 UTC m=+9.262285652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xq9rf" (UniqueName: "kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf") pod "network-check-target-gkcs9" (UID: "83eae010-1fa5-4861-ad26-9be01d46a741") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:54.061908 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:54.061826 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:54.062103 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:54.062003 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:57:55.064190 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:55.064144 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:57:55.064568 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:55.064398 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:57:55.067876 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:55.064942 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:57:55.067876 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:55.065114 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:57:56.061483 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:56.061449 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:56.061686 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:56.061597 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:57:57.061381 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:57.061348 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:57:57.061877 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:57.061348 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:57:57.061877 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:57.061498 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:57:57.061877 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:57.061558 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:57:57.739197 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:57.739156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs\") pod \"network-metrics-daemon-bkxp5\" (UID: \"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd\") " pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:57:57.739422 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:57.739213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:57.739422 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:57.739322 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:57.739422 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:57.739402 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs podName:a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd nodeName:}" failed. No retries permitted until 2026-09-06 13:58:05.739379847 +0000 UTC m=+17.194868708 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs") pod "network-metrics-daemon-bkxp5" (UID: "a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:57.739601 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:57.739322 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:57.739601 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:57.739504 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret podName:53540329-a02b-4fcb-ac80-8d6bf3489b10 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:05.739480017 +0000 UTC m=+17.194968874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret") pod "global-pull-secret-syncer-9fqd2" (UID: "53540329-a02b-4fcb-ac80-8d6bf3489b10") : object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:57.841195 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:57.840481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9rf\" (UniqueName: \"kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf\") pod \"network-check-target-gkcs9\" (UID: \"83eae010-1fa5-4861-ad26-9be01d46a741\") " pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:57:57.841195 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:57.840673 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 13:57:57.841195 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:57.840693 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 13:57:57.841195 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:57.840706 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xq9rf for pod openshift-network-diagnostics/network-check-target-gkcs9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:57.841195 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:57.840766 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf podName:83eae010-1fa5-4861-ad26-9be01d46a741 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:05.840746624 +0000 UTC m=+17.296235472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xq9rf" (UniqueName: "kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf") pod "network-check-target-gkcs9" (UID: "83eae010-1fa5-4861-ad26-9be01d46a741") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:58.061619 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:58.061538 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:57:58.062249 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:58.061686 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:57:59.062797 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:59.062749 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:57:59.063320 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:59.062876 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:57:59.063320 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:57:59.063292 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:57:59.063436 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:57:59.063396 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:00.061388 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:00.061351 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:00.061558 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:00.061479 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:01.061395 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:01.061363 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:01.061804 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:01.061489 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:01.061804 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:01.061555 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:01.061804 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:01.061663 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:02.062038 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:02.062007 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:02.062516 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:02.062138 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:03.061634 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:03.061598 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:03.061782 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:03.061614 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:03.061782 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:03.061731 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:03.061877 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:03.061793 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:04.061673 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:04.061640 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:04.062090 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:04.061750 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:05.061719 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:05.061685 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:05.061719 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:05.061717 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:05.062195 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:05.061832 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:05.062195 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:05.061982 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:05.799042 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:05.798998 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs\") pod \"network-metrics-daemon-bkxp5\" (UID: \"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd\") " pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:05.799250 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:05.799055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:05.799250 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:05.799118 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:58:05.799250 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:05.799190 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 13:58:05.799250 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:05.799197 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs podName:a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd nodeName:}" failed. No retries permitted until 2026-09-06 13:58:21.79917503 +0000 UTC m=+33.254663879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs") pod "network-metrics-daemon-bkxp5" (UID: "a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:58:05.799435 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:05.799260 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret podName:53540329-a02b-4fcb-ac80-8d6bf3489b10 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:21.799242861 +0000 UTC m=+33.254731709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret") pod "global-pull-secret-syncer-9fqd2" (UID: "53540329-a02b-4fcb-ac80-8d6bf3489b10") : object "kube-system"/"original-pull-secret" not registered Sep 06 13:58:05.899803 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:05.899774 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9rf\" (UniqueName: \"kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf\") pod \"network-check-target-gkcs9\" (UID: \"83eae010-1fa5-4861-ad26-9be01d46a741\") " pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:05.900005 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:05.899968 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 13:58:05.900005 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:05.899993 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 13:58:05.900005 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:05.900005 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xq9rf for pod openshift-network-diagnostics/network-check-target-gkcs9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:58:05.900172 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:05.900070 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf podName:83eae010-1fa5-4861-ad26-9be01d46a741 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:21.90004994 +0000 UTC m=+33.355538797 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xq9rf" (UniqueName: "kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf") pod "network-check-target-gkcs9" (UID: "83eae010-1fa5-4861-ad26-9be01d46a741") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:58:06.062082 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:06.062011 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:06.062504 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:06.062130 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:07.061142 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:07.061106 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:07.061323 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:07.061112 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:07.061323 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:07.061270 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:07.061440 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:07.061388 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:08.061587 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:08.061549 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:08.061987 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:08.061663 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:09.062093 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:09.062062 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:09.062533 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:09.062173 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:09.062533 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:09.062274 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:09.062533 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:09.062390 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:10.061391 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:10.061242 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:10.061475 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:10.061462 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:10.377441 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:10.377408 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" event={"ID":"c6753921-b1e1-433c-8065-f10c62e274f4","Type":"ContainerStarted","Data":"ce1c2d46b933838803a52eb630430942860ca8d5b1adaa1c74a85a609241ba4d"} Sep 06 13:58:10.386862 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:10.386072 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ddhdd" event={"ID":"c264c419-5ce3-43d4-b604-8580a52f3518","Type":"ContainerStarted","Data":"9ad426bdb5f97815ac82e864d3d2ff90270650d864e1ea4b246669383641a0de"} Sep 06 13:58:10.394293 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:10.394114 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-113.ec2.internal" event={"ID":"d4e8e4996366f70a783b63f04db7c684","Type":"ContainerStarted","Data":"ed12845bc46b5421b66b9b98b8fae54ef16325a7255265d424c72eb332fd3b34"} Sep 06 13:58:10.396121 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:10.395818 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p9w44" event={"ID":"68e4ea42-b401-4f77-844d-1eee77f1d8b0","Type":"ContainerStarted","Data":"dbcb7ceb2f1f3d0c5f93a14015dc76512638c383f7df8bd080033480c6c1acea"} Sep 06 13:58:10.402498 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:10.402397 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ddhdd" podStartSLOduration=1.663651298 podStartE2EDuration="21.402376643s" totalImagesPullingTime="19.738725345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:10.402091607 +0000 UTC m=+21.857580470" watchObservedRunningTime="2026-09-06 13:58:10.402376643 +0000 UTC m=+21.857865584" Sep 06 13:58:10.435626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:10.434444 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-113.ec2.internal" podStartSLOduration=20.434423604 podStartE2EDuration="20.434423604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:10.434186366 +0000 UTC m=+21.889675230" watchObservedRunningTime="2026-09-06 13:58:10.434423604 +0000 UTC m=+21.889912467" Sep 06 13:58:10.435626 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:10.435098 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p9w44" podStartSLOduration=1.79166415 podStartE2EDuration="21.435083197s" totalImagesPullingTime="19.643419047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:10.419170453 +0000 UTC m=+21.874659317" watchObservedRunningTime="2026-09-06 13:58:10.435083197 +0000 UTC m=+21.890572061" Sep 06 13:58:11.061321 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.061289 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:11.061467 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.061289 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:11.061467 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:11.061439 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:11.061559 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:11.061523 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:11.399222 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.399140 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-98p47" event={"ID":"82c12af7-6d5b-4f10-871c-442eea180684","Type":"ContainerStarted","Data":"417fa9d928e56372c75421385b9abdbf8dd48897c24eccc02e9645d32fb086f2"} Sep 06 13:58:11.400920 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.400886 2577 generic.go:356] "Generic (PLEG): container finished" podID="cdb2dbef-18c2-4a14-a421-f15c0286c411" containerID="dc35abcc90c6de450de45916851f8f00095d6bd18ac8cdd78e0c1d955e0ea312" exitCode=0 Sep 06 13:58:11.401013 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.400934 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7x44l" event={"ID":"cdb2dbef-18c2-4a14-a421-f15c0286c411","Type":"ContainerDied","Data":"dc35abcc90c6de450de45916851f8f00095d6bd18ac8cdd78e0c1d955e0ea312"} Sep 06 13:58:11.402528 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.402493 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" event={"ID":"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978","Type":"ContainerStarted","Data":"60c400fa0524aa03979573c8746e73a4bd702a424c1ea0d52e69598f69a353b7"} Sep 06 13:58:11.405959 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.405936 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g7hzl" event={"ID":"23a987fd-669e-4094-9025-a403d8be2a2d","Type":"ContainerStarted","Data":"6856d79eabe66ee62010970db0ed05ce7187b5bf9455fe829edf845cfaa5822e"} Sep 06 13:58:11.407254 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.407232 2577 generic.go:356] "Generic (PLEG): container finished" podID="88425535be0e3f6db9fbc0fc46a4e7fd" containerID="3129f16281214dad6f4c786c768a2de0259ad12a2a6248bced520ef157a6f078" exitCode=0 Sep 06 13:58:11.407350 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.407300 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal" event={"ID":"88425535be0e3f6db9fbc0fc46a4e7fd","Type":"ContainerDied","Data":"3129f16281214dad6f4c786c768a2de0259ad12a2a6248bced520ef157a6f078"} Sep 06 13:58:11.408614 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.408596 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ms5fd" event={"ID":"ab2f34a9-090c-4b27-a511-310e6d0e50ed","Type":"ContainerStarted","Data":"ddd03ec6d018cea0bf42a92b377ba4c204585bcba7a41a83a65a6c3323d017be"} Sep 06 13:58:11.411289 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.411266 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" event={"ID":"c6753921-b1e1-433c-8065-f10c62e274f4","Type":"ContainerStarted","Data":"089330d18d8a143aa694e8639ade90bad69f8401acd813b36eedaf6fa7c8612a"} Sep 06 13:58:11.411366 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.411337 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" event={"ID":"c6753921-b1e1-433c-8065-f10c62e274f4","Type":"ContainerStarted","Data":"5d9543c40fe2f25936383966678d103e0e02303464c581a1fef63ee2f64edb22"} Sep 06 13:58:11.411419 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.411386 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" event={"ID":"c6753921-b1e1-433c-8065-f10c62e274f4","Type":"ContainerStarted","Data":"5ab69f66ca08e2dd9f0a5fd72d5ab5bb21db1debf17c22544e5d8c9be4972031"} Sep 06 13:58:11.411462 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.411431 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" event={"ID":"c6753921-b1e1-433c-8065-f10c62e274f4","Type":"ContainerStarted","Data":"700b8ba20faf414db46a075df505fd4fdd4be821eb956737f221f7de3020e13f"} Sep 06 13:58:11.411498 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.411477 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" event={"ID":"c6753921-b1e1-433c-8065-f10c62e274f4","Type":"ContainerStarted","Data":"b5d4f4a62f35ebfa0eb3d03d39309c3ce09f289fd60bd130a315c76d8259ccb9"} Sep 06 13:58:11.411929 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.411879 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-98p47" podStartSLOduration=2.719313625 podStartE2EDuration="22.411864281s" totalImagesPullingTime="19.692550656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:11.411696633 +0000 UTC m=+22.867185496" watchObservedRunningTime="2026-09-06 13:58:11.411864281 +0000 UTC m=+22.867353147" Sep 06 13:58:11.412712 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.412686 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-42dlx" event={"ID":"61206fdd-09e3-449e-b320-c62e226616e0","Type":"ContainerStarted","Data":"c8c40b2a3e41a54db2822531f24142567b7dde5c1a6a8d6e957c8dcb66a4d30f"} Sep 06 13:58:11.440224 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.440192 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g7hzl" podStartSLOduration=2.7830798789999998 podStartE2EDuration="22.440181074s" totalImagesPullingTime="19.657101195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:11.423617575 +0000 UTC m=+22.879106438" watchObservedRunningTime="2026-09-06 13:58:11.440181074 +0000 UTC m=+22.895669936" Sep 06 13:58:11.465693 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.465656 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ms5fd" podStartSLOduration=6.73092374 podStartE2EDuration="22.465645931s" totalImagesPullingTime="15.734722191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:11.465400815 +0000 UTC m=+22.920889676" watchObservedRunningTime="2026-09-06 13:58:11.465645931 +0000 UTC m=+22.921134793" Sep 06 13:58:11.503991 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.503966 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:11.562969 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.562827 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:11.653446 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.653424 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 06 13:58:11.663179 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:11.663160 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:12.019627 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:12.019532 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-06T13:58:11.653441434Z","UUID":"0ecb56a8-44f8-484b-8240-fbeb56af27e1","Handler":null,"Name":"","Endpoint":""} Sep 06 13:58:12.021128 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:12.021111 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 06 13:58:12.021241 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:12.021141 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 06 13:58:12.061896 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:12.061870 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:12.062023 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:12.061971 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:12.418303 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:12.418219 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" event={"ID":"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978","Type":"ContainerStarted","Data":"191fbeae66191258d432cb70a31bef481968d7cd625a960a1804be070697bd2d"} Sep 06 13:58:12.420559 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:12.420473 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal" event={"ID":"88425535be0e3f6db9fbc0fc46a4e7fd","Type":"ContainerStarted","Data":"411c2a82afaaff8691f6ca161169836a7849aceb757a4fb1bee37e4b5a22ddbc"} Sep 06 13:58:12.435771 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:12.435145 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-42dlx" podStartSLOduration=7.621547722 podStartE2EDuration="23.43512809s" totalImagesPullingTime="15.813580368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:11.477335191 +0000 UTC m=+22.932824052" watchObservedRunningTime="2026-09-06 13:58:12.43512809 +0000 UTC m=+23.890616953" Sep 06 13:58:12.435931 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:12.435763 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-113.ec2.internal" podStartSLOduration=22.4357534 podStartE2EDuration="22.4357534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:12.434808554 +0000 UTC m=+23.890297415" watchObservedRunningTime="2026-09-06 13:58:12.4357534 +0000 UTC m=+23.891242266" Sep 06 13:58:13.062108 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:13.062081 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:13.062222 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:13.062084 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:13.062222 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:13.062194 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:13.062332 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:13.062311 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:13.147753 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:13.147722 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ms5fd" Sep 06 13:58:13.148476 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:13.148320 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ms5fd" Sep 06 13:58:13.228675 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:13.228605 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:13.328253 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:13.328216 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:13.425779 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:13.425745 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" event={"ID":"ff9cd7ae-4a5e-45ef-9cf5-52edb8424978","Type":"ContainerStarted","Data":"3eb7462850e9de2f0f5108c89cf8caee97e8118549284aa96222b4b038be1028"} Sep 06 13:58:13.429932 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:13.429877 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" event={"ID":"c6753921-b1e1-433c-8065-f10c62e274f4","Type":"ContainerStarted","Data":"3993e13e1fc7b96d5fed77a89a15fce008718e6b8eeb2699a65288f5344b3886"} Sep 06 13:58:13.443356 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:13.443316 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdz74" podStartSLOduration=2.911490535 podStartE2EDuration="24.443303007s" totalImagesPullingTime="21.531812472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:13.442813352 +0000 UTC m=+24.898302213" watchObservedRunningTime="2026-09-06 13:58:13.443303007 +0000 UTC m=+24.898791868" Sep 06 13:58:14.061980 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:14.061950 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:14.062147 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:14.062070 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:14.432558 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:14.432492 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 06 13:58:15.061225 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:15.061194 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:15.061410 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:15.061315 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:15.061471 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:15.061313 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:15.061471 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:15.061442 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:15.438687 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:15.438423 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" event={"ID":"c6753921-b1e1-433c-8065-f10c62e274f4","Type":"ContainerStarted","Data":"b428f20da4916272f04c2d8db626648e4ee9943b093375f428518cbe03eeb8be"} Sep 06 13:58:15.439560 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:15.438725 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:58:15.439560 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:15.438805 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:58:15.455178 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:15.455028 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:58:15.455178 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:15.455123 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:58:15.471770 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:15.471442 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" podStartSLOduration=6.560483663 podStartE2EDuration="26.471426331s" totalImagesPullingTime="19.910942668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:15.470326719 +0000 UTC m=+26.925815581" watchObservedRunningTime="2026-09-06 13:58:15.471426331 +0000 UTC m=+26.926915247" Sep 06 13:58:15.477444 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:15.477335 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:15.535928 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:15.535751 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:15.856734 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:15.856704 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:15.940445 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:15.940388 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:16.005362 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:16.005327 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:16.061759 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:16.061735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:16.061873 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:16.061850 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:16.429698 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:16.429674 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-42dlx_61206fdd-09e3-449e-b320-c62e226616e0/dns-node-resolver/0.log" Sep 06 13:58:16.441950 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:16.441923 2577 generic.go:356] "Generic (PLEG): container finished" podID="cdb2dbef-18c2-4a14-a421-f15c0286c411" containerID="b06800f7854777ce9ef39163e0786954e7369785bdbe6557ea3031f91c6a0d00" exitCode=0 Sep 06 13:58:16.442335 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:16.441995 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7x44l" event={"ID":"cdb2dbef-18c2-4a14-a421-f15c0286c411","Type":"ContainerDied","Data":"b06800f7854777ce9ef39163e0786954e7369785bdbe6557ea3031f91c6a0d00"} Sep 06 13:58:16.442335 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:16.442100 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 06 13:58:16.614417 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:16.614388 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:16.715524 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:16.715434 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:17.061401 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:17.061371 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:17.061566 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:17.061371 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:17.061566 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:17.061525 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:17.061648 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:17.061581 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:17.404880 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:17.404819 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9fqd2"] Sep 06 13:58:17.405019 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:17.404969 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:17.405100 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:17.405053 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:17.407775 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:17.407750 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gkcs9"] Sep 06 13:58:17.408448 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:17.408428 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bkxp5"] Sep 06 13:58:17.430917 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:17.430866 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-98p47_82c12af7-6d5b-4f10-871c-442eea180684/node-ca/0.log" Sep 06 13:58:17.445711 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:17.445691 2577 generic.go:356] "Generic (PLEG): container finished" podID="cdb2dbef-18c2-4a14-a421-f15c0286c411" containerID="de38e17961750a1543775fe18a1e1b27c5787982c6075d9cfd909ce3096b4300" exitCode=0 Sep 06 13:58:17.446308 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:17.445784 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7x44l" event={"ID":"cdb2dbef-18c2-4a14-a421-f15c0286c411","Type":"ContainerDied","Data":"de38e17961750a1543775fe18a1e1b27c5787982c6075d9cfd909ce3096b4300"} Sep 06 13:58:17.446308 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:17.445810 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:17.446308 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:17.445821 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:17.446308 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:17.445928 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 06 13:58:17.446308 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:17.445987 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:17.446564 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:17.446457 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:17.494179 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:17.494029 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:17.543579 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:17.543553 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:18.249111 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:18.249084 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:18.300630 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:18.300588 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:18.401311 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:18.401244 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:18.450278 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:18.450250 2577 generic.go:356] "Generic (PLEG): container finished" podID="cdb2dbef-18c2-4a14-a421-f15c0286c411" containerID="37967f03338059dc98a07e33568124d1d806b8b7dfeacb2188ac9367e706f904" exitCode=0 Sep 06 13:58:18.450622 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:18.450300 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7x44l" event={"ID":"cdb2dbef-18c2-4a14-a421-f15c0286c411","Type":"ContainerDied","Data":"37967f03338059dc98a07e33568124d1d806b8b7dfeacb2188ac9367e706f904"} Sep 06 13:58:18.471608 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:18.471584 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:18.610685 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:18.610653 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:18.667530 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:18.667444 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:19.049304 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:19.049276 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:19.062223 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:19.062199 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:19.062383 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:19.062305 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:19.062383 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:19.062382 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:19.062537 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:19.062465 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:19.062610 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:19.062526 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:19.062610 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:19.062539 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:19.549151 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:19.549121 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:58:20.086520 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:20.086478 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ms5fd" Sep 06 13:58:20.086710 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:20.086626 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 06 13:58:20.087229 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:20.087207 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ms5fd" Sep 06 13:58:21.061611 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:21.061582 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:21.062076 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:21.061582 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:21.062076 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:21.061709 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:21.062076 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:21.061789 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:21.062076 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:21.061582 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:21.062076 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:21.061922 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:21.825801 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:21.825772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs\") pod \"network-metrics-daemon-bkxp5\" (UID: \"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd\") " pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:21.825801 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:21.825813 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:21.826089 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:21.825947 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 13:58:21.826089 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:21.826008 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret podName:53540329-a02b-4fcb-ac80-8d6bf3489b10 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:53.82598824 +0000 UTC m=+65.281477080 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret") pod "global-pull-secret-syncer-9fqd2" (UID: "53540329-a02b-4fcb-ac80-8d6bf3489b10") : object "kube-system"/"original-pull-secret" not registered Sep 06 13:58:21.826089 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:21.825947 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:58:21.826089 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:21.826091 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs podName:a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd nodeName:}" failed. No retries permitted until 2026-09-06 13:58:53.826074938 +0000 UTC m=+65.281563794 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs") pod "network-metrics-daemon-bkxp5" (UID: "a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:58:21.926483 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:21.926410 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9rf\" (UniqueName: \"kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf\") pod \"network-check-target-gkcs9\" (UID: \"83eae010-1fa5-4861-ad26-9be01d46a741\") " pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:21.926636 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:21.926594 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 13:58:21.926636 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:21.926616 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 13:58:21.926636 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:21.926628 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xq9rf for pod openshift-network-diagnostics/network-check-target-gkcs9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:58:21.926790 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:21.926706 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf podName:83eae010-1fa5-4861-ad26-9be01d46a741 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:53.926669151 +0000 UTC m=+65.382157992 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xq9rf" (UniqueName: "kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf") pod "network-check-target-gkcs9" (UID: "83eae010-1fa5-4861-ad26-9be01d46a741") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:58:21.995617 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:21.995569 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:22.186100 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:22.186019 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:23.066840 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:23.066781 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:23.066958 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:23.066894 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:23.066958 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:23.066935 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:23.066958 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:23.066925 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:23.067120 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:23.067004 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:23.067120 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:23.067074 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:23.376277 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:23.376198 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:23.463694 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:23.463659 2577 generic.go:356] "Generic (PLEG): container finished" podID="cdb2dbef-18c2-4a14-a421-f15c0286c411" containerID="e93913611ba8cfa52202c3c04dd50a4b087ec8ea6e41dc0aa07585c6f1cb5a7e" exitCode=0 Sep 06 13:58:23.463825 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:23.463715 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7x44l" event={"ID":"cdb2dbef-18c2-4a14-a421-f15c0286c411","Type":"ContainerDied","Data":"e93913611ba8cfa52202c3c04dd50a4b087ec8ea6e41dc0aa07585c6f1cb5a7e"} Sep 06 13:58:23.478529 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:23.478498 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:23.576695 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:23.576666 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:23.675969 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:23.675946 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:23.777280 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:23.777248 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:23.877653 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:23.877621 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:23.978991 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:23.978921 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:24.141916 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:24.141876 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:24.468479 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:24.468454 2577 generic.go:356] "Generic (PLEG): container finished" podID="cdb2dbef-18c2-4a14-a421-f15c0286c411" containerID="9f4a5659330d68ce35537c6925bc02c11d0184f2a51450b22036d8c3fee257e7" exitCode=0 Sep 06 13:58:24.468819 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:24.468506 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7x44l" event={"ID":"cdb2dbef-18c2-4a14-a421-f15c0286c411","Type":"ContainerDied","Data":"9f4a5659330d68ce35537c6925bc02c11d0184f2a51450b22036d8c3fee257e7"} Sep 06 13:58:24.944438 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:24.944265 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:25.064453 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:25.064432 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:25.064602 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:25.064539 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:25.064665 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:25.064627 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:25.064746 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:25.064729 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:25.064806 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:25.064798 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:25.064899 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:25.064883 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:25.469462 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:25.469428 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:25.473449 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:25.473423 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7x44l" event={"ID":"cdb2dbef-18c2-4a14-a421-f15c0286c411","Type":"ContainerStarted","Data":"db2c2d79ea4743695b4ca67118f813daade2bc3c1520454068219ee9fe4f9e23"} Sep 06 13:58:25.494792 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:25.494739 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7x44l" podStartSLOduration=6.059823536 podStartE2EDuration="35.494718802s" totalImagesPullingTime="29.434895266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:25.492674825 +0000 UTC m=+36.948163687" watchObservedRunningTime="2026-09-06 13:58:25.494718802 +0000 UTC m=+36.950207665" Sep 06 13:58:25.570357 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:25.570331 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:25.734679 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:25.734612 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:25.940318 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:25.940294 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:26.334335 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:26.334298 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:26.932128 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:26.932104 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:27.061939 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:27.061898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:27.062063 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:27.061952 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:27.062063 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:27.061924 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:27.062063 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:27.062035 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:27.062163 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:27.062107 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:27.062197 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:27.062170 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:27.134632 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:27.134604 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:27.337971 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:27.337946 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:27.389089 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:27.389068 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:27.734713 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:27.734687 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:28.134831 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:28.134800 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:28.233616 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:28.233575 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:28.335799 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:28.335767 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:28.538079 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:28.538053 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:29.063937 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:29.062995 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:29.063937 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:29.063394 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:29.063937 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:29.063661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:29.063937 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:29.063776 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:29.065282 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:29.064321 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:29.065282 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:29.064439 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:29.728956 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:29.728921 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:29.830291 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:29.830259 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:29.938441 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:29.938411 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:30.132021 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:30.131987 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:30.335160 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:30.335128 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:30.534047 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:30.534021 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:31.062066 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:31.062038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:31.062066 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:31.062055 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:31.062483 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:31.062046 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:31.062483 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:31.062149 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:31.062483 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:31.062206 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:31.062483 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:31.062264 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:31.335828 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:31.335748 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:31.537520 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:31.537487 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:31.935556 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:31.935526 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:32.732619 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:32.732586 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:32.931415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:32.931386 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:33.064282 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:33.064242 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:33.064480 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:33.064404 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:33.064627 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:33.064242 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:33.064889 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:33.064867 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:33.064996 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:33.064607 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:33.065311 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:33.065292 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:33.137037 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:33.137008 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:33.532389 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:33.532356 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:33.934531 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:33.934457 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:34.137177 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:34.137144 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:35.061253 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:35.061223 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:35.061644 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:35.061223 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:35.061644 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:35.061328 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:35.061644 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:35.061384 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:35.061644 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:35.061232 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:35.061644 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:35.061444 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:36.875710 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:36.875680 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:36.932060 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:36.932020 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:37.061169 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:37.061133 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:37.061169 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:37.061159 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:37.061392 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:37.061232 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:37.061392 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:37.061246 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:37.061392 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:37.061310 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:37.061392 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:37.061378 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:38.367494 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:38.367441 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:38.418250 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:38.418222 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:39.062699 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.062669 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:39.062860 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:39.062744 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkcs9" podUID="83eae010-1fa5-4861-ad26-9be01d46a741" Sep 06 13:58:39.062860 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.062845 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:39.062997 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:39.062977 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fqd2" podUID="53540329-a02b-4fcb-ac80-8d6bf3489b10" Sep 06 13:58:39.063060 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.063046 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:39.063184 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:39.063166 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkxp5" podUID="a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd" Sep 06 13:58:39.147591 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.147562 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:39.238396 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.238364 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:39.328532 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.328468 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:39.429031 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.429010 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:39.809880 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.809851 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-113.ec2.internal" event="NodeReady" Sep 06 13:58:39.809999 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.809965 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 06 13:58:39.858195 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.858167 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lj66p"] Sep 06 13:58:39.860206 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.860191 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cq6wh"] Sep 06 13:58:39.860318 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.860303 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:39.862025 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.862008 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:39.862718 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.862698 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 06 13:58:39.862819 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.862809 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mvgg4\"" Sep 06 13:58:39.862885 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.862833 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 06 13:58:39.864277 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.864261 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 06 13:58:39.864462 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.864424 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 06 13:58:39.864533 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.864480 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 06 13:58:39.864622 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.864607 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-58qh8\"" Sep 06 13:58:39.864879 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.864866 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 06 13:58:39.869336 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.869318 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lj66p"] Sep 06 13:58:39.872439 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.872412 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cq6wh"] Sep 06 13:58:39.958466 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.958382 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49733091-cbf3-4000-a5c2-388553158a1b-metrics-tls\") pod \"dns-default-lj66p\" (UID: \"49733091-cbf3-4000-a5c2-388553158a1b\") " pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:39.958466 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.958427 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49733091-cbf3-4000-a5c2-388553158a1b-tmp-dir\") pod \"dns-default-lj66p\" (UID: \"49733091-cbf3-4000-a5c2-388553158a1b\") " pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:39.958466 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.958456 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ndnn\" (UniqueName: \"kubernetes.io/projected/49733091-cbf3-4000-a5c2-388553158a1b-kube-api-access-9ndnn\") pod \"dns-default-lj66p\" (UID: \"49733091-cbf3-4000-a5c2-388553158a1b\") " pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:39.958719 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.958513 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49733091-cbf3-4000-a5c2-388553158a1b-config-volume\") pod \"dns-default-lj66p\" (UID: \"49733091-cbf3-4000-a5c2-388553158a1b\") " pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:39.958719 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.958592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/791ee34c-1b29-496a-b728-f253cd243af0-data-volume\") pod \"insights-runtime-extractor-cq6wh\" (UID: \"791ee34c-1b29-496a-b728-f253cd243af0\") " pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:39.958719 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.958625 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/791ee34c-1b29-496a-b728-f253cd243af0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq6wh\" (UID: \"791ee34c-1b29-496a-b728-f253cd243af0\") " pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:39.958719 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.958643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcwgj\" (UniqueName: \"kubernetes.io/projected/791ee34c-1b29-496a-b728-f253cd243af0-kube-api-access-vcwgj\") pod \"insights-runtime-extractor-cq6wh\" (UID: \"791ee34c-1b29-496a-b728-f253cd243af0\") " pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:39.958719 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.958660 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/791ee34c-1b29-496a-b728-f253cd243af0-crio-socket\") pod \"insights-runtime-extractor-cq6wh\" (UID: \"791ee34c-1b29-496a-b728-f253cd243af0\") " pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:39.958719 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.958678 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/791ee34c-1b29-496a-b728-f253cd243af0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq6wh\" (UID: \"791ee34c-1b29-496a-b728-f253cd243af0\") " pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:39.960016 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.959996 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kxgfz"] Sep 06 13:58:39.961724 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.961710 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kxgfz" Sep 06 13:58:39.964349 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.964307 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dt8p5\"" Sep 06 13:58:39.964349 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.964320 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 06 13:58:39.964349 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.964335 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 06 13:58:39.964531 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.964313 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 06 13:58:39.971012 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.970994 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kxgfz"] Sep 06 13:58:39.980548 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:39.980526 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:40.059794 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.059771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49733091-cbf3-4000-a5c2-388553158a1b-metrics-tls\") pod \"dns-default-lj66p\" (UID: \"49733091-cbf3-4000-a5c2-388553158a1b\") " pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:40.059960 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.059811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49733091-cbf3-4000-a5c2-388553158a1b-tmp-dir\") pod \"dns-default-lj66p\" (UID: \"49733091-cbf3-4000-a5c2-388553158a1b\") " pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:40.059960 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.059839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ndnn\" (UniqueName: \"kubernetes.io/projected/49733091-cbf3-4000-a5c2-388553158a1b-kube-api-access-9ndnn\") pod \"dns-default-lj66p\" (UID: \"49733091-cbf3-4000-a5c2-388553158a1b\") " pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:40.059960 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.059893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49733091-cbf3-4000-a5c2-388553158a1b-config-volume\") pod \"dns-default-lj66p\" (UID: \"49733091-cbf3-4000-a5c2-388553158a1b\") " pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:40.059960 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.059930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0865f16e-e175-49f7-a333-8a59f3380eb7-cert\") pod \"ingress-canary-kxgfz\" (UID: \"0865f16e-e175-49f7-a333-8a59f3380eb7\") " pod="openshift-ingress-canary/ingress-canary-kxgfz" Sep 06 13:58:40.060176 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.059964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/791ee34c-1b29-496a-b728-f253cd243af0-data-volume\") pod \"insights-runtime-extractor-cq6wh\" (UID: \"791ee34c-1b29-496a-b728-f253cd243af0\") " pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:40.060176 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.060007 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/791ee34c-1b29-496a-b728-f253cd243af0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq6wh\" (UID: \"791ee34c-1b29-496a-b728-f253cd243af0\") " pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:40.060176 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.060031 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwgj\" (UniqueName: \"kubernetes.io/projected/791ee34c-1b29-496a-b728-f253cd243af0-kube-api-access-vcwgj\") pod \"insights-runtime-extractor-cq6wh\" (UID: \"791ee34c-1b29-496a-b728-f253cd243af0\") " pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:40.060176 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.060055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/791ee34c-1b29-496a-b728-f253cd243af0-crio-socket\") pod \"insights-runtime-extractor-cq6wh\" (UID: \"791ee34c-1b29-496a-b728-f253cd243af0\") " pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:40.060176 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.060075 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/791ee34c-1b29-496a-b728-f253cd243af0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq6wh\" (UID: \"791ee34c-1b29-496a-b728-f253cd243af0\") " pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:40.060176 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.060099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z76nc\" (UniqueName: \"kubernetes.io/projected/0865f16e-e175-49f7-a333-8a59f3380eb7-kube-api-access-z76nc\") pod \"ingress-canary-kxgfz\" (UID: \"0865f16e-e175-49f7-a333-8a59f3380eb7\") " pod="openshift-ingress-canary/ingress-canary-kxgfz" Sep 06 13:58:40.060382 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.060208 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49733091-cbf3-4000-a5c2-388553158a1b-tmp-dir\") pod \"dns-default-lj66p\" (UID: \"49733091-cbf3-4000-a5c2-388553158a1b\") " pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:40.060382 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.060224 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/791ee34c-1b29-496a-b728-f253cd243af0-crio-socket\") pod \"insights-runtime-extractor-cq6wh\" (UID: \"791ee34c-1b29-496a-b728-f253cd243af0\") " pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:40.060382 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.060305 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/791ee34c-1b29-496a-b728-f253cd243af0-data-volume\") pod \"insights-runtime-extractor-cq6wh\" (UID: \"791ee34c-1b29-496a-b728-f253cd243af0\") " pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:40.060707 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.060686 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/791ee34c-1b29-496a-b728-f253cd243af0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq6wh\" (UID: \"791ee34c-1b29-496a-b728-f253cd243af0\") " pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:40.060759 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.060686 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49733091-cbf3-4000-a5c2-388553158a1b-config-volume\") pod \"dns-default-lj66p\" (UID: \"49733091-cbf3-4000-a5c2-388553158a1b\") " pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:40.063838 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.063815 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49733091-cbf3-4000-a5c2-388553158a1b-metrics-tls\") pod \"dns-default-lj66p\" (UID: \"49733091-cbf3-4000-a5c2-388553158a1b\") " pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:40.063838 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.063828 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/791ee34c-1b29-496a-b728-f253cd243af0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq6wh\" (UID: \"791ee34c-1b29-496a-b728-f253cd243af0\") " pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:40.069706 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.069686 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcwgj\" (UniqueName: \"kubernetes.io/projected/791ee34c-1b29-496a-b728-f253cd243af0-kube-api-access-vcwgj\") pod \"insights-runtime-extractor-cq6wh\" (UID: \"791ee34c-1b29-496a-b728-f253cd243af0\") " pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:40.069788 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.069723 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ndnn\" (UniqueName: \"kubernetes.io/projected/49733091-cbf3-4000-a5c2-388553158a1b-kube-api-access-9ndnn\") pod \"dns-default-lj66p\" (UID: \"49733091-cbf3-4000-a5c2-388553158a1b\") " pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:40.098529 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.098482 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:40.161074 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.161055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0865f16e-e175-49f7-a333-8a59f3380eb7-cert\") pod \"ingress-canary-kxgfz\" (UID: \"0865f16e-e175-49f7-a333-8a59f3380eb7\") " pod="openshift-ingress-canary/ingress-canary-kxgfz" Sep 06 13:58:40.161130 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.161090 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z76nc\" (UniqueName: \"kubernetes.io/projected/0865f16e-e175-49f7-a333-8a59f3380eb7-kube-api-access-z76nc\") pod \"ingress-canary-kxgfz\" (UID: \"0865f16e-e175-49f7-a333-8a59f3380eb7\") " pod="openshift-ingress-canary/ingress-canary-kxgfz" Sep 06 13:58:40.163303 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.163278 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0865f16e-e175-49f7-a333-8a59f3380eb7-cert\") pod \"ingress-canary-kxgfz\" (UID: \"0865f16e-e175-49f7-a333-8a59f3380eb7\") " pod="openshift-ingress-canary/ingress-canary-kxgfz" Sep 06 13:58:40.168234 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.168208 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z76nc\" (UniqueName: \"kubernetes.io/projected/0865f16e-e175-49f7-a333-8a59f3380eb7-kube-api-access-z76nc\") pod \"ingress-canary-kxgfz\" (UID: \"0865f16e-e175-49f7-a333-8a59f3380eb7\") " pod="openshift-ingress-canary/ingress-canary-kxgfz" Sep 06 13:58:40.170849 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.170826 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:40.177456 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.177364 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cq6wh" Sep 06 13:58:40.270253 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.270230 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kxgfz" Sep 06 13:58:40.449881 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.449852 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lj66p"] Sep 06 13:58:40.452752 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.452730 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cq6wh"] Sep 06 13:58:40.453847 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:40.453741 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49733091_cbf3_4000_a5c2_388553158a1b.slice/crio-fe9e93ccafb22d6640448b7732bf4aabb845f6188f2258cb9dbe0f830a409db5 WatchSource:0}: Error finding container fe9e93ccafb22d6640448b7732bf4aabb845f6188f2258cb9dbe0f830a409db5: Status 404 returned error can't find the container with id fe9e93ccafb22d6640448b7732bf4aabb845f6188f2258cb9dbe0f830a409db5 Sep 06 13:58:40.455695 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:40.455674 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791ee34c_1b29_496a_b728_f253cd243af0.slice/crio-8884dc74c5714392b52dbe9d6365109f59dbfdb3493c957dc92a24a33d8898c4 WatchSource:0}: Error finding container 8884dc74c5714392b52dbe9d6365109f59dbfdb3493c957dc92a24a33d8898c4: Status 404 returned error can't find the container with id 8884dc74c5714392b52dbe9d6365109f59dbfdb3493c957dc92a24a33d8898c4 Sep 06 13:58:40.498178 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.498155 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:40.504615 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.504593 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lj66p" event={"ID":"49733091-cbf3-4000-a5c2-388553158a1b","Type":"ContainerStarted","Data":"fe9e93ccafb22d6640448b7732bf4aabb845f6188f2258cb9dbe0f830a409db5"} Sep 06 13:58:40.505693 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.505663 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq6wh" event={"ID":"791ee34c-1b29-496a-b728-f253cd243af0","Type":"ContainerStarted","Data":"8884dc74c5714392b52dbe9d6365109f59dbfdb3493c957dc92a24a33d8898c4"} Sep 06 13:58:40.537274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.537251 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kxgfz"] Sep 06 13:58:40.540043 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:40.540024 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0865f16e_e175_49f7_a333_8a59f3380eb7.slice/crio-836eb2c9a8bcfc8a7df82e6e634d020773f36dc99904e5b922f6d6895465fa9c WatchSource:0}: Error finding container 836eb2c9a8bcfc8a7df82e6e634d020773f36dc99904e5b922f6d6895465fa9c: Status 404 returned error can't find the container with id 836eb2c9a8bcfc8a7df82e6e634d020773f36dc99904e5b922f6d6895465fa9c Sep 06 13:58:40.598285 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:40.598257 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:41.067025 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:41.066994 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:41.067343 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:41.067319 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:41.067724 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:41.067705 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:41.070415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:41.070395 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-htlhn\"" Sep 06 13:58:41.070539 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:41.070430 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 06 13:58:41.070539 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:41.070440 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8cxw7\"" Sep 06 13:58:41.070845 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:41.070828 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 06 13:58:41.070938 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:41.070883 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 06 13:58:41.071011 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:41.070829 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 06 13:58:41.510591 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:41.510470 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq6wh" event={"ID":"791ee34c-1b29-496a-b728-f253cd243af0","Type":"ContainerStarted","Data":"eaab970b4bcf60f200cbf3f1f942df5b92e663a88371af4726cee36402eb3187"} Sep 06 13:58:41.510591 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:41.510541 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq6wh" event={"ID":"791ee34c-1b29-496a-b728-f253cd243af0","Type":"ContainerStarted","Data":"c77f8884151731751453df5e396be85db8e9c9ec2cc2bdfe48eb38e7e5398ed3"} Sep 06 13:58:41.511829 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:41.511793 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kxgfz" event={"ID":"0865f16e-e175-49f7-a333-8a59f3380eb7","Type":"ContainerStarted","Data":"836eb2c9a8bcfc8a7df82e6e634d020773f36dc99904e5b922f6d6895465fa9c"} Sep 06 13:58:42.668766 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:42.668723 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:42.726195 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:42.726164 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:42.816184 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:42.816156 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:42.818883 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:42.818857 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-86c2s"] Sep 06 13:58:42.820707 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:42.820687 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-86c2s" Sep 06 13:58:42.822883 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:42.822863 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2x9z8\"" Sep 06 13:58:42.823129 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:42.823009 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 06 13:58:42.829242 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:42.829225 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-86c2s"] Sep 06 13:58:42.882336 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:42.882309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/552fa68e-cd7a-473c-a685-b49178b6686b-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-86c2s\" (UID: \"552fa68e-cd7a-473c-a685-b49178b6686b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-86c2s" Sep 06 13:58:42.969085 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:42.969003 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:42.983002 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:42.982975 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/552fa68e-cd7a-473c-a685-b49178b6686b-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-86c2s\" (UID: \"552fa68e-cd7a-473c-a685-b49178b6686b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-86c2s" Sep 06 13:58:42.983160 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:42.983140 2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 06 13:58:42.983215 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:42.983207 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/552fa68e-cd7a-473c-a685-b49178b6686b-tls-certificates podName:552fa68e-cd7a-473c-a685-b49178b6686b nodeName:}" failed. No retries permitted until 2026-09-06 13:58:43.483188276 +0000 UTC m=+54.938677129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/552fa68e-cd7a-473c-a685-b49178b6686b-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-86c2s" (UID: "552fa68e-cd7a-473c-a685-b49178b6686b") : secret "prometheus-operator-admission-webhook-tls" not found Sep 06 13:58:43.068942 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:43.068896 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:43.171515 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:43.171477 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:43.488004 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:43.487923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/552fa68e-cd7a-473c-a685-b49178b6686b-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-86c2s\" (UID: \"552fa68e-cd7a-473c-a685-b49178b6686b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-86c2s" Sep 06 13:58:43.490264 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:43.490233 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/552fa68e-cd7a-473c-a685-b49178b6686b-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-86c2s\" (UID: \"552fa68e-cd7a-473c-a685-b49178b6686b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-86c2s" Sep 06 13:58:43.518730 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:43.518695 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lj66p" event={"ID":"49733091-cbf3-4000-a5c2-388553158a1b","Type":"ContainerStarted","Data":"08b17a17a5f74f4384811f13f1afca72d2f503e65ebcf86c289b49f38daabc5c"} Sep 06 13:58:43.518867 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:43.518771 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lj66p" event={"ID":"49733091-cbf3-4000-a5c2-388553158a1b","Type":"ContainerStarted","Data":"22b113aa30720552634226c7524ea8a21edddfe89c2e854bbad82e9af3d50c44"} Sep 06 13:58:43.518867 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:43.518801 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:43.522419 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:43.521347 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq6wh" event={"ID":"791ee34c-1b29-496a-b728-f253cd243af0","Type":"ContainerStarted","Data":"6277ccf5ab9d5938dde114b1eda5a3ba22a679f0cd653dec420469f3734a4335"} Sep 06 13:58:43.524045 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:43.524020 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kxgfz" event={"ID":"0865f16e-e175-49f7-a333-8a59f3380eb7","Type":"ContainerStarted","Data":"3fcf2a4c31d077c5bc0b4c0c25e22b0efbc2db93284f03065b6123e10b23e250"} Sep 06 13:58:43.533945 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:43.533878 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lj66p" podStartSLOduration=1.794229058 podStartE2EDuration="4.533860768s" totalImagesPullingTime="2.73963171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:43.533661317 +0000 UTC m=+54.989150180" watchObservedRunningTime="2026-09-06 13:58:43.533860768 +0000 UTC m=+54.989349632" Sep 06 13:58:43.549237 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:43.549200 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cq6wh" podStartSLOduration=1.950851731 podStartE2EDuration="4.549187298s" totalImagesPullingTime="2.598335567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:58:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:43.548519623 +0000 UTC m=+55.004008486" watchObservedRunningTime="2026-09-06 13:58:43.549187298 +0000 UTC m=+55.004676157" Sep 06 13:58:43.561813 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:43.561775 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kxgfz" podStartSLOduration=1.901396168 podStartE2EDuration="4.561763855s" totalImagesPullingTime="2.660367687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:43.561262462 +0000 UTC m=+55.016751323" watchObservedRunningTime="2026-09-06 13:58:43.561763855 +0000 UTC m=+55.017252716" Sep 06 13:58:43.730436 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:43.730401 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-86c2s" Sep 06 13:58:43.995265 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:43.995193 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-86c2s"] Sep 06 13:58:43.998308 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:43.998283 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552fa68e_cd7a_473c_a685_b49178b6686b.slice/crio-d2cbb6bbd4585d2f05875360602935d80be2cb9ba6a4709a851b8fd16a983da0 WatchSource:0}: Error finding container d2cbb6bbd4585d2f05875360602935d80be2cb9ba6a4709a851b8fd16a983da0: Status 404 returned error can't find the container with id d2cbb6bbd4585d2f05875360602935d80be2cb9ba6a4709a851b8fd16a983da0 Sep 06 13:58:44.528427 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:44.528388 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-86c2s" event={"ID":"552fa68e-cd7a-473c-a685-b49178b6686b","Type":"ContainerStarted","Data":"d2cbb6bbd4585d2f05875360602935d80be2cb9ba6a4709a851b8fd16a983da0"} Sep 06 13:58:45.536786 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:45.536750 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-86c2s" event={"ID":"552fa68e-cd7a-473c-a685-b49178b6686b","Type":"ContainerStarted","Data":"dc398d60533ef8c316ebe6a89a0be2fdd5dc4acd15d11e5c5362a1daadffb978"} Sep 06 13:58:45.537191 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:45.536950 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-86c2s" Sep 06 13:58:45.541727 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:45.541705 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-86c2s" Sep 06 13:58:45.551599 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:45.551544 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-86c2s" podStartSLOduration=2.683691681 podStartE2EDuration="3.551532284s" totalImagesPullingTime="867.840603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:45.55117153 +0000 UTC m=+57.006660394" watchObservedRunningTime="2026-09-06 13:58:45.551532284 +0000 UTC m=+57.007021146" Sep 06 13:58:45.837640 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:45.837551 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:45.868237 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:45.868212 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-55d5f"] Sep 06 13:58:45.894103 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:45.894074 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-55d5f"] Sep 06 13:58:45.894221 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:45.894169 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" Sep 06 13:58:45.896232 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:45.896212 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bsmxp\"" Sep 06 13:58:45.897182 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:45.897144 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 06 13:58:45.897182 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:45.897147 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 06 13:58:45.897182 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:45.897147 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 06 13:58:45.897417 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:45.897397 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 06 13:58:45.897483 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:45.897456 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 06 13:58:45.912466 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:45.912441 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:46.008274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:46.008251 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a6639dd-bc29-4b35-8b48-595283323be3-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-55d5f\" (UID: \"9a6639dd-bc29-4b35-8b48-595283323be3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" Sep 06 13:58:46.008374 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:46.008287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzvr9\" (UniqueName: \"kubernetes.io/projected/9a6639dd-bc29-4b35-8b48-595283323be3-kube-api-access-fzvr9\") pod \"prometheus-operator-6f4946db56-55d5f\" (UID: \"9a6639dd-bc29-4b35-8b48-595283323be3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" Sep 06 13:58:46.008374 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:46.008310 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a6639dd-bc29-4b35-8b48-595283323be3-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-55d5f\" (UID: \"9a6639dd-bc29-4b35-8b48-595283323be3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" Sep 06 13:58:46.008374 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:46.008366 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a6639dd-bc29-4b35-8b48-595283323be3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-55d5f\" (UID: \"9a6639dd-bc29-4b35-8b48-595283323be3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" Sep 06 13:58:46.109066 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:46.108988 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a6639dd-bc29-4b35-8b48-595283323be3-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-55d5f\" (UID: \"9a6639dd-bc29-4b35-8b48-595283323be3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" Sep 06 13:58:46.109066 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:46.109032 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzvr9\" (UniqueName: \"kubernetes.io/projected/9a6639dd-bc29-4b35-8b48-595283323be3-kube-api-access-fzvr9\") pod \"prometheus-operator-6f4946db56-55d5f\" (UID: \"9a6639dd-bc29-4b35-8b48-595283323be3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" Sep 06 13:58:46.109066 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:46.109053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a6639dd-bc29-4b35-8b48-595283323be3-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-55d5f\" (UID: \"9a6639dd-bc29-4b35-8b48-595283323be3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" Sep 06 13:58:46.109291 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:46.109153 2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 06 13:58:46.109291 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:46.109219 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6639dd-bc29-4b35-8b48-595283323be3-prometheus-operator-tls podName:9a6639dd-bc29-4b35-8b48-595283323be3 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:46.609200301 +0000 UTC m=+58.064689154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/9a6639dd-bc29-4b35-8b48-595283323be3-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-55d5f" (UID: "9a6639dd-bc29-4b35-8b48-595283323be3") : secret "prometheus-operator-tls" not found Sep 06 13:58:46.109291 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:46.109158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a6639dd-bc29-4b35-8b48-595283323be3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-55d5f\" (UID: \"9a6639dd-bc29-4b35-8b48-595283323be3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" Sep 06 13:58:46.109730 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:46.109713 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a6639dd-bc29-4b35-8b48-595283323be3-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-55d5f\" (UID: \"9a6639dd-bc29-4b35-8b48-595283323be3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" Sep 06 13:58:46.111475 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:46.111454 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a6639dd-bc29-4b35-8b48-595283323be3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-55d5f\" (UID: \"9a6639dd-bc29-4b35-8b48-595283323be3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" Sep 06 13:58:46.118851 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:46.118829 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzvr9\" (UniqueName: \"kubernetes.io/projected/9a6639dd-bc29-4b35-8b48-595283323be3-kube-api-access-fzvr9\") pod \"prometheus-operator-6f4946db56-55d5f\" (UID: \"9a6639dd-bc29-4b35-8b48-595283323be3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" Sep 06 13:58:46.410312 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:46.410222 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:46.612592 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:46.612560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a6639dd-bc29-4b35-8b48-595283323be3-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-55d5f\" (UID: \"9a6639dd-bc29-4b35-8b48-595283323be3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" Sep 06 13:58:46.614830 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:46.614806 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a6639dd-bc29-4b35-8b48-595283323be3-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-55d5f\" (UID: \"9a6639dd-bc29-4b35-8b48-595283323be3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" Sep 06 13:58:46.802245 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:46.802210 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" Sep 06 13:58:47.092172 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:47.092093 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-55d5f"] Sep 06 13:58:47.095061 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:47.095037 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6639dd_bc29_4b35_8b48_595283323be3.slice/crio-b120d516bca5bf25ce20ecb8ecc36ab9c2f62e5988c9903f757e6804689efa7c WatchSource:0}: Error finding container b120d516bca5bf25ce20ecb8ecc36ab9c2f62e5988c9903f757e6804689efa7c: Status 404 returned error can't find the container with id b120d516bca5bf25ce20ecb8ecc36ab9c2f62e5988c9903f757e6804689efa7c Sep 06 13:58:47.545228 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:47.545188 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" event={"ID":"9a6639dd-bc29-4b35-8b48-595283323be3","Type":"ContainerStarted","Data":"b120d516bca5bf25ce20ecb8ecc36ab9c2f62e5988c9903f757e6804689efa7c"} Sep 06 13:58:47.809724 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:47.809634 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:47.862702 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:47.862669 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:48.479245 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:48.479203 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:48.550155 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:48.550119 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" event={"ID":"9a6639dd-bc29-4b35-8b48-595283323be3","Type":"ContainerStarted","Data":"8645a783e24a7a00dd4b7a2c873e0075eaaa43c9a5f2571ba5c967d877528d82"} Sep 06 13:58:48.550318 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:48.550180 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" event={"ID":"9a6639dd-bc29-4b35-8b48-595283323be3","Type":"ContainerStarted","Data":"f2dea7333ae0af4339f7d8232c0ef68ce2b374fc71537d0a720f602da7f7c49f"} Sep 06 13:58:48.568841 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:48.568794 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-55d5f" podStartSLOduration=2.425177255 podStartE2EDuration="3.568776813s" totalImagesPullingTime="1.143599558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:48.567112722 +0000 UTC m=+60.022601584" watchObservedRunningTime="2026-09-06 13:58:48.568776813 +0000 UTC m=+60.024265713" Sep 06 13:58:49.223022 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.222990 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-4h9mc"] Sep 06 13:58:49.225652 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.225636 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-4h9mc" Sep 06 13:58:49.227927 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.227894 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 06 13:58:49.228070 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.228049 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 06 13:58:49.228136 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.228063 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mfwqn\"" Sep 06 13:58:49.229441 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.229423 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:49.234106 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.234087 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-4h9mc"] Sep 06 13:58:49.282505 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.282482 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:49.332677 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.332652 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qmts\" (UniqueName: \"kubernetes.io/projected/c7c35f62-5d92-4fd0-82b4-cc6274ecd0c1-kube-api-access-8qmts\") pod \"downloads-85d98c998f-4h9mc\" (UID: \"c7c35f62-5d92-4fd0-82b4-cc6274ecd0c1\") " pod="openshift-console/downloads-85d98c998f-4h9mc" Sep 06 13:58:49.433638 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.433608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qmts\" (UniqueName: \"kubernetes.io/projected/c7c35f62-5d92-4fd0-82b4-cc6274ecd0c1-kube-api-access-8qmts\") pod \"downloads-85d98c998f-4h9mc\" (UID: \"c7c35f62-5d92-4fd0-82b4-cc6274ecd0c1\") " pod="openshift-console/downloads-85d98c998f-4h9mc" Sep 06 13:58:49.443956 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.443929 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qmts\" (UniqueName: \"kubernetes.io/projected/c7c35f62-5d92-4fd0-82b4-cc6274ecd0c1-kube-api-access-8qmts\") pod \"downloads-85d98c998f-4h9mc\" (UID: \"c7c35f62-5d92-4fd0-82b4-cc6274ecd0c1\") " pod="openshift-console/downloads-85d98c998f-4h9mc" Sep 06 13:58:49.456166 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.456144 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:49.534089 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.534066 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-4h9mc" Sep 06 13:58:49.559839 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.559818 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwnd7" Sep 06 13:58:49.669991 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.669960 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:49.727043 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.727015 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:49.776896 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.776867 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:49.798099 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.798037 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-4h9mc"] Sep 06 13:58:49.801402 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:49.801377 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c35f62_5d92_4fd0_82b4_cc6274ecd0c1.slice/crio-4c19d75f389c1fe88bb2fa369142a3590e70156db51123af3f5878451a254ec8 WatchSource:0}: Error finding container 4c19d75f389c1fe88bb2fa369142a3590e70156db51123af3f5878451a254ec8: Status 404 returned error can't find the container with id 4c19d75f389c1fe88bb2fa369142a3590e70156db51123af3f5878451a254ec8 Sep 06 13:58:49.841177 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.841153 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:49.943384 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:49.943353 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:50.040848 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.040818 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:50.193518 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.193440 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:50.199736 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.199706 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt"] Sep 06 13:58:50.202878 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.202862 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" Sep 06 13:58:50.205210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.205184 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 06 13:58:50.205338 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.205220 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 06 13:58:50.205338 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.205249 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jjvlt\"" Sep 06 13:58:50.212350 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.212328 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt"] Sep 06 13:58:50.220746 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.220719 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9wsfj"] Sep 06 13:58:50.223872 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.223856 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.226189 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.226168 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 06 13:58:50.226299 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.226205 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 06 13:58:50.226299 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.226247 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zxjvq\"" Sep 06 13:58:50.226299 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.226252 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 06 13:58:50.230694 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.230676 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-7qp65"] Sep 06 13:58:50.232825 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.232810 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.235327 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.235308 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pl697\"" Sep 06 13:58:50.235421 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.235327 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 06 13:58:50.235421 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.235354 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 06 13:58:50.235421 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.235308 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 06 13:58:50.240987 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.240968 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f2faa94-8b19-46ae-9c4b-0fb3b8de760a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-jfkwt\" (UID: \"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" Sep 06 13:58:50.241066 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.240993 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddfqc\" (UniqueName: \"kubernetes.io/projected/9f2faa94-8b19-46ae-9c4b-0fb3b8de760a-kube-api-access-ddfqc\") pod \"openshift-state-metrics-657fd6c54-jfkwt\" (UID: \"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" Sep 06 13:58:50.241066 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.241027 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f2faa94-8b19-46ae-9c4b-0fb3b8de760a-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-jfkwt\" (UID: \"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" Sep 06 13:58:50.241131 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.241107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f2faa94-8b19-46ae-9c4b-0fb3b8de760a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-jfkwt\" (UID: \"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" Sep 06 13:58:50.247270 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.247248 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-7qp65"] Sep 06 13:58:50.273825 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.273798 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:50.333168 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.333139 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:50.341717 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.341694 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b157e249-7489-4dd2-bd2b-47f85605cd9d-metrics-client-ca\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.341807 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.341741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f2faa94-8b19-46ae-9c4b-0fb3b8de760a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-jfkwt\" (UID: \"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" Sep 06 13:58:50.341807 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.341770 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.341807 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.341797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3ec859a2-4603-4f11-809d-83112426a10d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.341963 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.341831 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ec859a2-4603-4f11-809d-83112426a10d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.342017 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.341957 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f2faa94-8b19-46ae-9c4b-0fb3b8de760a-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-jfkwt\" (UID: \"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" Sep 06 13:58:50.342017 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.341987 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-textfile\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.342017 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.342008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec859a2-4603-4f11-809d-83112426a10d-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.342127 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.342038 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ec859a2-4603-4f11-809d-83112426a10d-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.342127 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.342063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-tls\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.342127 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.342082 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94hs6\" (UniqueName: \"kubernetes.io/projected/b157e249-7489-4dd2-bd2b-47f85605cd9d-kube-api-access-94hs6\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.342218 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.342155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f2faa94-8b19-46ae-9c4b-0fb3b8de760a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-jfkwt\" (UID: \"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" Sep 06 13:58:50.342218 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.342175 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddfqc\" (UniqueName: \"kubernetes.io/projected/9f2faa94-8b19-46ae-9c4b-0fb3b8de760a-kube-api-access-ddfqc\") pod \"openshift-state-metrics-657fd6c54-jfkwt\" (UID: \"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" Sep 06 13:58:50.342218 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.342198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-wtmp\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.342218 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.342214 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.342352 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.342238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b157e249-7489-4dd2-bd2b-47f85605cd9d-sys\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.342352 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.342288 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3ec859a2-4603-4f11-809d-83112426a10d-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.342352 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:50.342309 2577 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 06 13:58:50.342352 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.342336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m7sh\" (UniqueName: \"kubernetes.io/projected/3ec859a2-4603-4f11-809d-83112426a10d-kube-api-access-8m7sh\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.342496 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:50.342407 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2faa94-8b19-46ae-9c4b-0fb3b8de760a-openshift-state-metrics-tls podName:9f2faa94-8b19-46ae-9c4b-0fb3b8de760a nodeName:}" failed. No retries permitted until 2026-09-06 13:58:50.842388048 +0000 UTC m=+62.297876895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9f2faa94-8b19-46ae-9c4b-0fb3b8de760a-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-jfkwt" (UID: "9f2faa94-8b19-46ae-9c4b-0fb3b8de760a") : secret "openshift-state-metrics-tls" not found Sep 06 13:58:50.342496 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.342449 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b157e249-7489-4dd2-bd2b-47f85605cd9d-root\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.342729 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.342714 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f2faa94-8b19-46ae-9c4b-0fb3b8de760a-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-jfkwt\" (UID: \"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" Sep 06 13:58:50.344100 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.344083 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f2faa94-8b19-46ae-9c4b-0fb3b8de760a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-jfkwt\" (UID: \"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" Sep 06 13:58:50.356441 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.356418 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddfqc\" (UniqueName: \"kubernetes.io/projected/9f2faa94-8b19-46ae-9c4b-0fb3b8de760a-kube-api-access-ddfqc\") pod \"openshift-state-metrics-657fd6c54-jfkwt\" (UID: \"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" Sep 06 13:58:50.433208 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.433182 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:50.443360 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443335 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b157e249-7489-4dd2-bd2b-47f85605cd9d-metrics-client-ca\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.443473 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.443473 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3ec859a2-4603-4f11-809d-83112426a10d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.443473 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443434 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ec859a2-4603-4f11-809d-83112426a10d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.443655 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443492 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-textfile\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.443655 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443523 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec859a2-4603-4f11-809d-83112426a10d-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.443655 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ec859a2-4603-4f11-809d-83112426a10d-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.443655 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443589 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-tls\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.443655 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443613 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94hs6\" (UniqueName: \"kubernetes.io/projected/b157e249-7489-4dd2-bd2b-47f85605cd9d-kube-api-access-94hs6\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.443929 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-wtmp\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.443929 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443700 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.443929 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443729 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b157e249-7489-4dd2-bd2b-47f85605cd9d-sys\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.443929 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443756 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3ec859a2-4603-4f11-809d-83112426a10d-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.443929 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m7sh\" (UniqueName: \"kubernetes.io/projected/3ec859a2-4603-4f11-809d-83112426a10d-kube-api-access-8m7sh\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.443929 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b157e249-7489-4dd2-bd2b-47f85605cd9d-root\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.444274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443938 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b157e249-7489-4dd2-bd2b-47f85605cd9d-root\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.444274 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.443988 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b157e249-7489-4dd2-bd2b-47f85605cd9d-metrics-client-ca\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.444274 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:50.444099 2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 06 13:58:50.444274 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:50.444154 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-tls podName:b157e249-7489-4dd2-bd2b-47f85605cd9d nodeName:}" failed. No retries permitted until 2026-09-06 13:58:50.944135604 +0000 UTC m=+62.399624450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-tls") pod "node-exporter-9wsfj" (UID: "b157e249-7489-4dd2-bd2b-47f85605cd9d") : secret "node-exporter-tls" not found Sep 06 13:58:50.444477 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.444433 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.444743 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.444720 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-wtmp\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.445050 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.445025 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3ec859a2-4603-4f11-809d-83112426a10d-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.445234 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.445044 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3ec859a2-4603-4f11-809d-83112426a10d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.445327 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.445052 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-textfile\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.445327 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.445101 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b157e249-7489-4dd2-bd2b-47f85605cd9d-sys\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.445327 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:50.445115 2577 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 06 13:58:50.445467 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:50.445386 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ec859a2-4603-4f11-809d-83112426a10d-kube-state-metrics-tls podName:3ec859a2-4603-4f11-809d-83112426a10d nodeName:}" failed. No retries permitted until 2026-09-06 13:58:50.945352606 +0000 UTC m=+62.400841463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3ec859a2-4603-4f11-809d-83112426a10d-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-7qp65" (UID: "3ec859a2-4603-4f11-809d-83112426a10d") : secret "kube-state-metrics-tls" not found Sep 06 13:58:50.445539 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.445519 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec859a2-4603-4f11-809d-83112426a10d-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.447447 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.447420 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.447590 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.447570 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ec859a2-4603-4f11-809d-83112426a10d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.454764 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.454726 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m7sh\" (UniqueName: \"kubernetes.io/projected/3ec859a2-4603-4f11-809d-83112426a10d-kube-api-access-8m7sh\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.454862 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.454816 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94hs6\" (UniqueName: \"kubernetes.io/projected/b157e249-7489-4dd2-bd2b-47f85605cd9d-kube-api-access-94hs6\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.537409 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.537367 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:50.557978 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.557897 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-4h9mc" event={"ID":"c7c35f62-5d92-4fd0-82b4-cc6274ecd0c1","Type":"ContainerStarted","Data":"4c19d75f389c1fe88bb2fa369142a3590e70156db51123af3f5878451a254ec8"} Sep 06 13:58:50.634061 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.634022 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:50.735498 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.735422 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:50.834933 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.834865 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:50.847176 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.847148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f2faa94-8b19-46ae-9c4b-0fb3b8de760a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-jfkwt\" (UID: \"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" Sep 06 13:58:50.849663 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.849640 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f2faa94-8b19-46ae-9c4b-0fb3b8de760a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-jfkwt\" (UID: \"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" Sep 06 13:58:50.934338 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.934312 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:50.948515 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.948484 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ec859a2-4603-4f11-809d-83112426a10d-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:50.948515 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.948517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-tls\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.951159 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.951132 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b157e249-7489-4dd2-bd2b-47f85605cd9d-node-exporter-tls\") pod \"node-exporter-9wsfj\" (UID: \"b157e249-7489-4dd2-bd2b-47f85605cd9d\") " pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:50.951268 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:50.951239 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ec859a2-4603-4f11-809d-83112426a10d-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-7qp65\" (UID: \"3ec859a2-4603-4f11-809d-83112426a10d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:51.112223 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.112195 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" Sep 06 13:58:51.120084 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.120054 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:51.132346 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.132322 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9wsfj" Sep 06 13:58:51.140894 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.140870 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" Sep 06 13:58:51.142711 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:51.142686 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb157e249_7489_4dd2_bd2b_47f85605cd9d.slice/crio-4241ae45c523f23aae5389bc2f72abcd0f553ac30c13fe298783d4426e1978ba WatchSource:0}: Error finding container 4241ae45c523f23aae5389bc2f72abcd0f553ac30c13fe298783d4426e1978ba: Status 404 returned error can't find the container with id 4241ae45c523f23aae5389bc2f72abcd0f553ac30c13fe298783d4426e1978ba Sep 06 13:58:51.230641 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.230515 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:51.284769 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.284736 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 13:58:51.289844 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.289818 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.292032 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.292007 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 06 13:58:51.292159 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.292008 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 06 13:58:51.292221 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.292207 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 06 13:58:51.292574 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.292335 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 06 13:58:51.292574 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.292372 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 06 13:58:51.292574 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.292437 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 06 13:58:51.292574 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.292454 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 06 13:58:51.292574 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.292506 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 06 13:58:51.292574 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.292519 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xknm6\"" Sep 06 13:58:51.292574 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.292528 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 06 13:58:51.300791 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.300758 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 13:58:51.352304 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.352270 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.352304 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.352306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.352505 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.352340 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.352505 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.352397 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.352505 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.352458 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.352505 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.352504 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-config-out\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.352654 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.352526 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.352654 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.352575 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.352654 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.352609 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.352654 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.352631 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.352654 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.352648 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-web-config\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.352808 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.352669 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.352808 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.352685 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccq8z\" (UniqueName: \"kubernetes.io/projected/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-kube-api-access-ccq8z\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.390250 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.390153 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt"] Sep 06 13:58:51.394959 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:51.394930 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2faa94_8b19_46ae_9c4b_0fb3b8de760a.slice/crio-42123a5a7b3b35ea1cddd7bdd84349d5d967076c9e4c6df44c7367780bf4f932 WatchSource:0}: Error finding container 42123a5a7b3b35ea1cddd7bdd84349d5d967076c9e4c6df44c7367780bf4f932: Status 404 returned error can't find the container with id 42123a5a7b3b35ea1cddd7bdd84349d5d967076c9e4c6df44c7367780bf4f932 Sep 06 13:58:51.423645 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.423618 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-7qp65"] Sep 06 13:58:51.429333 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.429305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:51.429790 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:51.429764 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec859a2_4603_4f11_809d_83112426a10d.slice/crio-4bf00677ae231c0fc6ae8c579f53e0b8248201578839dea15b03e259113262cf WatchSource:0}: Error finding container 4bf00677ae231c0fc6ae8c579f53e0b8248201578839dea15b03e259113262cf: Status 404 returned error can't find the container with id 4bf00677ae231c0fc6ae8c579f53e0b8248201578839dea15b03e259113262cf Sep 06 13:58:51.453957 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.453932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.454075 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.453992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.454075 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.454025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.454075 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.454069 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.454216 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.454113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-config-out\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.454216 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.454146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.454216 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.454185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.455070 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.454688 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.455070 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.455018 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.455070 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.455051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.455297 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.455080 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-web-config\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.455297 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.455117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.455297 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.455144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccq8z\" (UniqueName: \"kubernetes.io/projected/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-kube-api-access-ccq8z\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.455297 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:51.455169 2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 06 13:58:51.455297 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.455183 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.455297 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:51.455232 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-main-tls podName:3ba37183-e831-4d29-82bd-9c9cb9a7ba52 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:51.955210644 +0000 UTC m=+63.410699498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "3ba37183-e831-4d29-82bd-9c9cb9a7ba52") : secret "alertmanager-main-tls" not found Sep 06 13:58:51.455715 ip-10-0-132-113 kubenswrapper[2577]: E0906 13:58:51.455347 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-alertmanager-trusted-ca-bundle podName:3ba37183-e831-4d29-82bd-9c9cb9a7ba52 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:51.955332829 +0000 UTC m=+63.410821669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "3ba37183-e831-4d29-82bd-9c9cb9a7ba52") : configmap references non-existent config key: ca-bundle.crt Sep 06 13:58:51.456120 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.456094 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.456884 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.456860 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.457944 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.457922 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.458340 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.458316 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.459100 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.458733 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.459100 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.458815 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.459100 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.458882 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-config-out\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.459100 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.459063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.459974 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.459934 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-web-config\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.464630 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.464596 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccq8z\" (UniqueName: \"kubernetes.io/projected/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-kube-api-access-ccq8z\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.563224 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.563181 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wsfj" event={"ID":"b157e249-7489-4dd2-bd2b-47f85605cd9d","Type":"ContainerStarted","Data":"4241ae45c523f23aae5389bc2f72abcd0f553ac30c13fe298783d4426e1978ba"} Sep 06 13:58:51.566765 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.566697 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" event={"ID":"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a","Type":"ContainerStarted","Data":"8575060a334518c59245b1cbd26e1981774d75ddc641f24d96b890c2aa1fc5ea"} Sep 06 13:58:51.566765 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.566754 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" event={"ID":"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a","Type":"ContainerStarted","Data":"0e3daf72b7444df7e886ca75ae60eb0a6216911ad989f0643fa777495978a455"} Sep 06 13:58:51.567111 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.566782 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" event={"ID":"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a","Type":"ContainerStarted","Data":"42123a5a7b3b35ea1cddd7bdd84349d5d967076c9e4c6df44c7367780bf4f932"} Sep 06 13:58:51.568538 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.568477 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" event={"ID":"3ec859a2-4603-4f11-809d-83112426a10d","Type":"ContainerStarted","Data":"4bf00677ae231c0fc6ae8c579f53e0b8248201578839dea15b03e259113262cf"} Sep 06 13:58:51.685455 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.685377 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:51.829797 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.829713 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:51.960138 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.960046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.960307 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.960172 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.961293 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.961260 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:51.970119 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:51.970076 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:52.132504 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.132433 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:52.201490 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.201355 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 06 13:58:52.220803 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.218856 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7876d5bf99-mn6rr"] Sep 06 13:58:52.224297 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.222249 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.227998 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.226985 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 06 13:58:52.227998 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.227219 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 06 13:58:52.227998 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.227364 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 06 13:58:52.227998 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.227694 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 06 13:58:52.227998 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.227781 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 06 13:58:52.227998 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.227873 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-q67fg\"" Sep 06 13:58:52.227998 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.227936 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-40hmqpmvo33p0\"" Sep 06 13:58:52.257623 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.257520 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7876d5bf99-mn6rr"] Sep 06 13:58:52.260370 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.260177 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:52.263787 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.263260 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.263787 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.263331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eb9bded-0cf1-41ab-b338-87538c1644a6-metrics-client-ca\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.263787 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.263364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brf4t\" (UniqueName: \"kubernetes.io/projected/9eb9bded-0cf1-41ab-b338-87538c1644a6-kube-api-access-brf4t\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.263787 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.263394 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-grpc-tls\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.263787 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.263419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-thanos-querier-tls\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.263787 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.263444 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.263787 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.263468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.263787 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.263519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.364689 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.364265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eb9bded-0cf1-41ab-b338-87538c1644a6-metrics-client-ca\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.364689 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.364317 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brf4t\" (UniqueName: \"kubernetes.io/projected/9eb9bded-0cf1-41ab-b338-87538c1644a6-kube-api-access-brf4t\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.364689 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.364346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-grpc-tls\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.364689 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.364373 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-thanos-querier-tls\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.364689 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.364409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.364689 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.364435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.364689 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.364487 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.365175 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.364830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.365175 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.365145 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eb9bded-0cf1-41ab-b338-87538c1644a6-metrics-client-ca\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.368432 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.368313 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.368432 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.368401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-grpc-tls\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.368673 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.368501 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.369310 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.369216 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-thanos-querier-tls\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.369310 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.369281 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.369686 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.369664 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9eb9bded-0cf1-41ab-b338-87538c1644a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.372405 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.372364 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brf4t\" (UniqueName: \"kubernetes.io/projected/9eb9bded-0cf1-41ab-b338-87538c1644a6-kube-api-access-brf4t\") pod \"thanos-querier-7876d5bf99-mn6rr\" (UID: \"9eb9bded-0cf1-41ab-b338-87538c1644a6\") " pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.498320 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.498200 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 13:58:52.539816 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.539782 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:58:52.574696 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.574656 2577 generic.go:356] "Generic (PLEG): container finished" podID="b157e249-7489-4dd2-bd2b-47f85605cd9d" containerID="e5b5cd756eaa874efe56cec23ec8c1ec6dcf8aada023a6133689bd0142b8804a" exitCode=0 Sep 06 13:58:52.574881 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.574753 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wsfj" event={"ID":"b157e249-7489-4dd2-bd2b-47f85605cd9d","Type":"ContainerDied","Data":"e5b5cd756eaa874efe56cec23ec8c1ec6dcf8aada023a6133689bd0142b8804a"} Sep 06 13:58:52.734015 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.733986 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:52.834497 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:52.834463 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:53.072112 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:53.072081 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba37183_e831_4d29_82bd_9c9cb9a7ba52.slice/crio-30a56a62b2c817d944ae709cd048f27cae6da7b94002bd54eae0020fe64e398b WatchSource:0}: Error finding container 30a56a62b2c817d944ae709cd048f27cae6da7b94002bd54eae0020fe64e398b: Status 404 returned error can't find the container with id 30a56a62b2c817d944ae709cd048f27cae6da7b94002bd54eae0020fe64e398b Sep 06 13:58:53.158121 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.158041 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:53.210347 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.210312 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:53.478088 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.478058 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:53.530669 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.530643 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lj66p" Sep 06 13:58:53.583661 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.583565 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wsfj" event={"ID":"b157e249-7489-4dd2-bd2b-47f85605cd9d","Type":"ContainerStarted","Data":"e5adef9e5245b44ef96c762b8f7eb4b1ef14a47579380c2f09a1c42659c7a868"} Sep 06 13:58:53.592575 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.592500 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" event={"ID":"3ec859a2-4603-4f11-809d-83112426a10d","Type":"ContainerStarted","Data":"f51575d5fc3f315b0505ee9ef17f12710884e584ea35d0a24516af40ff3e48af"} Sep 06 13:58:53.594463 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.594437 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerStarted","Data":"30a56a62b2c817d944ae709cd048f27cae6da7b94002bd54eae0020fe64e398b"} Sep 06 13:58:53.727543 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.726810 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7876d5bf99-mn6rr"] Sep 06 13:58:53.730185 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:53.730156 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb9bded_0cf1_41ab_b338_87538c1644a6.slice/crio-33e8a7ce7aadf206ac1765b6eff145cce5ba258535ce9597f2b73c3146b68bd3 WatchSource:0}: Error finding container 33e8a7ce7aadf206ac1765b6eff145cce5ba258535ce9597f2b73c3146b68bd3: Status 404 returned error can't find the container with id 33e8a7ce7aadf206ac1765b6eff145cce5ba258535ce9597f2b73c3146b68bd3 Sep 06 13:58:53.879347 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.878709 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs\") pod \"network-metrics-daemon-bkxp5\" (UID: \"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd\") " pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:53.879347 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.878818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:53.881090 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.880879 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 06 13:58:53.882236 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.881471 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 06 13:58:53.892431 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.892402 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53540329-a02b-4fcb-ac80-8d6bf3489b10-original-pull-secret\") pod \"global-pull-secret-syncer-9fqd2\" (UID: \"53540329-a02b-4fcb-ac80-8d6bf3489b10\") " pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:53.893330 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.893254 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd-metrics-certs\") pod \"network-metrics-daemon-bkxp5\" (UID: \"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd\") " pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:53.905395 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.905362 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:53.979963 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.979885 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9rf\" (UniqueName: \"kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf\") pod \"network-check-target-gkcs9\" (UID: \"83eae010-1fa5-4861-ad26-9be01d46a741\") " pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:53.982628 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.982603 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 06 13:58:53.992506 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.992479 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 06 13:58:53.994287 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:53.994268 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8cxw7\"" Sep 06 13:58:54.003717 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.003505 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkxp5" Sep 06 13:58:54.004625 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.004579 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9rf\" (UniqueName: \"kubernetes.io/projected/83eae010-1fa5-4861-ad26-9be01d46a741-kube-api-access-xq9rf\") pod \"network-check-target-gkcs9\" (UID: \"83eae010-1fa5-4861-ad26-9be01d46a741\") " pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:54.008333 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.008286 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fqd2" Sep 06 13:58:54.303817 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.303762 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-htlhn\"" Sep 06 13:58:54.307629 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.307593 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bkxp5"] Sep 06 13:58:54.312224 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.312202 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:58:54.334784 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.334757 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9fqd2"] Sep 06 13:58:54.468212 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:54.468175 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53540329_a02b_4fcb_ac80_8d6bf3489b10.slice/crio-225a17914abb8bc3bc8645c35da0be157dc0c27b46bc2bffe4eb847426cbf4cd WatchSource:0}: Error finding container 225a17914abb8bc3bc8645c35da0be157dc0c27b46bc2bffe4eb847426cbf4cd: Status 404 returned error can't find the container with id 225a17914abb8bc3bc8645c35da0be157dc0c27b46bc2bffe4eb847426cbf4cd Sep 06 13:58:54.469093 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:54.469066 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d5497c_1b54_4bb9_8a57_aa2ee736e7fd.slice/crio-7b4c14d744695d3384fa39a404b7312ae386ed945a60e468ebce18d342f4e86d WatchSource:0}: Error finding container 7b4c14d744695d3384fa39a404b7312ae386ed945a60e468ebce18d342f4e86d: Status 404 returned error can't find the container with id 7b4c14d744695d3384fa39a404b7312ae386ed945a60e468ebce18d342f4e86d Sep 06 13:58:54.604036 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.603321 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" event={"ID":"3ec859a2-4603-4f11-809d-83112426a10d","Type":"ContainerStarted","Data":"8a291296f5689eca313dc0e06f052b999182f6d62d607d684f337dc9cc4db1b1"} Sep 06 13:58:54.604036 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.603386 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" event={"ID":"3ec859a2-4603-4f11-809d-83112426a10d","Type":"ContainerStarted","Data":"0ee1868b973b5be276a207ef4bf56ab27d48c48f559631a131b79800eafa55b6"} Sep 06 13:58:54.607571 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.607052 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerStarted","Data":"5fe0fe7f5346bec0d564615124b4d3e3df3889288a5dda81bfda1884581ebc16"} Sep 06 13:58:54.608370 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.608343 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" event={"ID":"9eb9bded-0cf1-41ab-b338-87538c1644a6","Type":"ContainerStarted","Data":"33e8a7ce7aadf206ac1765b6eff145cce5ba258535ce9597f2b73c3146b68bd3"} Sep 06 13:58:54.613748 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.612680 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wsfj" event={"ID":"b157e249-7489-4dd2-bd2b-47f85605cd9d","Type":"ContainerStarted","Data":"77f85738d3eba5716b5454af615d5a5a6751b7dec650ac34c011a88cb5d1af82"} Sep 06 13:58:54.613748 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.612752 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6996bf4777-kh87z"] Sep 06 13:58:54.615744 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.615717 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.616843 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.616801 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" event={"ID":"9f2faa94-8b19-46ae-9c4b-0fb3b8de760a","Type":"ContainerStarted","Data":"870721d59d43b6c27ec0ad592d91c1034c7f3520e2fb1cf3f5ae18b11c5b5b27"} Sep 06 13:58:54.618111 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.618093 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1dbakejnp8fma\"" Sep 06 13:58:54.618188 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.618115 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 06 13:58:54.618429 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.618324 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-n2zjg\"" Sep 06 13:58:54.618429 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.618346 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 06 13:58:54.618805 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.618784 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 06 13:58:54.618889 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.618830 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 06 13:58:54.618957 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.618944 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9fqd2" event={"ID":"53540329-a02b-4fcb-ac80-8d6bf3489b10","Type":"ContainerStarted","Data":"225a17914abb8bc3bc8645c35da0be157dc0c27b46bc2bffe4eb847426cbf4cd"} Sep 06 13:58:54.621481 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.621431 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bkxp5" event={"ID":"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd","Type":"ContainerStarted","Data":"7b4c14d744695d3384fa39a404b7312ae386ed945a60e468ebce18d342f4e86d"} Sep 06 13:58:54.628978 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.627588 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-7qp65" podStartSLOduration=2.609409073 podStartE2EDuration="4.627570335s" totalImagesPullingTime="2.018161262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:54.625175357 +0000 UTC m=+66.080664225" watchObservedRunningTime="2026-09-06 13:58:54.627570335 +0000 UTC m=+66.083059198" Sep 06 13:58:54.629154 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.629135 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6996bf4777-kh87z"] Sep 06 13:58:54.665814 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.663847 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jfkwt" podStartSLOduration=2.744483394 podStartE2EDuration="4.663828272s" totalImagesPullingTime="1.919344878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:54.646547283 +0000 UTC m=+66.102036167" watchObservedRunningTime="2026-09-06 13:58:54.663828272 +0000 UTC m=+66.119317134" Sep 06 13:58:54.665814 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.664052 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9wsfj" podStartSLOduration=3.602181041 podStartE2EDuration="4.664042229s" totalImagesPullingTime="960.953552ms" totalInitContainerRuntime="100.907636ms" isStatefulPod=true podCreationTimestamp="2026-09-06 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:54.661667606 +0000 UTC m=+66.117156496" watchObservedRunningTime="2026-09-06 13:58:54.664042229 +0000 UTC m=+66.119531087" Sep 06 13:58:54.685980 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.685950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e080444a-1143-4e22-bcbe-b2180243a549-client-ca-bundle\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.686331 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.686311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e080444a-1143-4e22-bcbe-b2180243a549-secret-metrics-server-tls\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.686543 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.686515 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e080444a-1143-4e22-bcbe-b2180243a549-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.687406 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.686806 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e080444a-1143-4e22-bcbe-b2180243a549-secret-metrics-server-client-certs\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.687406 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.686957 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e080444a-1143-4e22-bcbe-b2180243a549-audit-log\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.687406 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.687003 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e080444a-1143-4e22-bcbe-b2180243a549-metrics-server-audit-profiles\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.687406 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.687030 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqcpz\" (UniqueName: \"kubernetes.io/projected/e080444a-1143-4e22-bcbe-b2180243a549-kube-api-access-tqcpz\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.748149 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.748120 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gkcs9"] Sep 06 13:58:54.752093 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:54.752062 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83eae010_1fa5_4861_ad26_9be01d46a741.slice/crio-ee5308a9768e3165ae0d6579a2c08da785492486a503014b1f117b3c6c6531d1 WatchSource:0}: Error finding container ee5308a9768e3165ae0d6579a2c08da785492486a503014b1f117b3c6c6531d1: Status 404 returned error can't find the container with id ee5308a9768e3165ae0d6579a2c08da785492486a503014b1f117b3c6c6531d1 Sep 06 13:58:54.788081 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.788029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e080444a-1143-4e22-bcbe-b2180243a549-audit-log\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.788081 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.788077 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e080444a-1143-4e22-bcbe-b2180243a549-metrics-server-audit-profiles\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.788259 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.788107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqcpz\" (UniqueName: \"kubernetes.io/projected/e080444a-1143-4e22-bcbe-b2180243a549-kube-api-access-tqcpz\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.788259 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.788240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e080444a-1143-4e22-bcbe-b2180243a549-client-ca-bundle\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.788365 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.788300 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e080444a-1143-4e22-bcbe-b2180243a549-secret-metrics-server-tls\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.788365 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.788346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e080444a-1143-4e22-bcbe-b2180243a549-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.788472 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.788401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e080444a-1143-4e22-bcbe-b2180243a549-secret-metrics-server-client-certs\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.788633 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.788526 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e080444a-1143-4e22-bcbe-b2180243a549-audit-log\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.789244 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.789198 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e080444a-1143-4e22-bcbe-b2180243a549-metrics-server-audit-profiles\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.789734 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.789665 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e080444a-1143-4e22-bcbe-b2180243a549-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.791995 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.791971 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e080444a-1143-4e22-bcbe-b2180243a549-secret-metrics-server-client-certs\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.792130 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.792106 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e080444a-1143-4e22-bcbe-b2180243a549-secret-metrics-server-tls\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.792354 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.792315 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e080444a-1143-4e22-bcbe-b2180243a549-client-ca-bundle\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.797185 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.797161 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqcpz\" (UniqueName: \"kubernetes.io/projected/e080444a-1143-4e22-bcbe-b2180243a549-kube-api-access-tqcpz\") pod \"metrics-server-6996bf4777-kh87z\" (UID: \"e080444a-1143-4e22-bcbe-b2180243a549\") " pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.962868 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.962782 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:58:54.991146 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.991113 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:54.991689 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.991665 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5"] Sep 06 13:58:54.995231 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.995209 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5" Sep 06 13:58:54.998541 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.997579 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lcnkz\"" Sep 06 13:58:54.998541 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:54.997590 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 06 13:58:55.006754 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.006685 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5"] Sep 06 13:58:55.087646 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.087560 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:55.091450 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.091307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b4f0419-be6f-4900-8116-6fdbe7278997-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-hjbv5\" (UID: \"3b4f0419-be6f-4900-8116-6fdbe7278997\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5" Sep 06 13:58:55.091450 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.091392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xzc6\" (UniqueName: \"kubernetes.io/projected/3b4f0419-be6f-4900-8116-6fdbe7278997-kube-api-access-2xzc6\") pod \"monitoring-plugin-85bbbbbf4-hjbv5\" (UID: \"3b4f0419-be6f-4900-8116-6fdbe7278997\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5" Sep 06 13:58:55.194037 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.193158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b4f0419-be6f-4900-8116-6fdbe7278997-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-hjbv5\" (UID: \"3b4f0419-be6f-4900-8116-6fdbe7278997\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5" Sep 06 13:58:55.194037 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.193257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xzc6\" (UniqueName: \"kubernetes.io/projected/3b4f0419-be6f-4900-8116-6fdbe7278997-kube-api-access-2xzc6\") pod \"monitoring-plugin-85bbbbbf4-hjbv5\" (UID: \"3b4f0419-be6f-4900-8116-6fdbe7278997\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5" Sep 06 13:58:55.196544 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.196442 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:55.202879 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.202829 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xzc6\" (UniqueName: \"kubernetes.io/projected/3b4f0419-be6f-4900-8116-6fdbe7278997-kube-api-access-2xzc6\") pod \"monitoring-plugin-85bbbbbf4-hjbv5\" (UID: \"3b4f0419-be6f-4900-8116-6fdbe7278997\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5" Sep 06 13:58:55.204134 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.204083 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b4f0419-be6f-4900-8116-6fdbe7278997-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-hjbv5\" (UID: \"3b4f0419-be6f-4900-8116-6fdbe7278997\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5" Sep 06 13:58:55.268202 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.268085 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6996bf4777-kh87z"] Sep 06 13:58:55.274115 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:55.274073 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode080444a_1143_4e22_bcbe_b2180243a549.slice/crio-847b8233050a18fa36120a62f57b1760d9e65ac9eacae71f3f28b9c1dbc14b34 WatchSource:0}: Error finding container 847b8233050a18fa36120a62f57b1760d9e65ac9eacae71f3f28b9c1dbc14b34: Status 404 returned error can't find the container with id 847b8233050a18fa36120a62f57b1760d9e65ac9eacae71f3f28b9c1dbc14b34 Sep 06 13:58:55.316006 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.315941 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5" Sep 06 13:58:55.606593 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.606505 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5"] Sep 06 13:58:55.628464 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.628432 2577 generic.go:356] "Generic (PLEG): container finished" podID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerID="5fe0fe7f5346bec0d564615124b4d3e3df3889288a5dda81bfda1884581ebc16" exitCode=0 Sep 06 13:58:55.628599 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.628521 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerDied","Data":"5fe0fe7f5346bec0d564615124b4d3e3df3889288a5dda81bfda1884581ebc16"} Sep 06 13:58:55.632832 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.632808 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" event={"ID":"e080444a-1143-4e22-bcbe-b2180243a549","Type":"ContainerStarted","Data":"847b8233050a18fa36120a62f57b1760d9e65ac9eacae71f3f28b9c1dbc14b34"} Sep 06 13:58:55.635140 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:55.635061 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gkcs9" event={"ID":"83eae010-1fa5-4861-ad26-9be01d46a741","Type":"ContainerStarted","Data":"ee5308a9768e3165ae0d6579a2c08da785492486a503014b1f117b3c6c6531d1"} Sep 06 13:58:56.006510 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.006390 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:56.486832 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.486764 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:56.490566 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.490536 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 13:58:56.494277 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.494257 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.497866 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.496579 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 06 13:58:56.497866 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.496894 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 06 13:58:56.497866 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.497183 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 06 13:58:56.497866 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.497363 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 06 13:58:56.497866 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.497514 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 06 13:58:56.498190 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.498029 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 06 13:58:56.498295 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.498272 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 06 13:58:56.501978 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.500941 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9fq4mll5g11dm\"" Sep 06 13:58:56.501978 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.501149 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7q8vk\"" Sep 06 13:58:56.502631 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.502438 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 06 13:58:56.502631 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.502499 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 06 13:58:56.503066 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.502813 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 06 13:58:56.503066 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.502889 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 06 13:58:56.503066 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.503022 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 06 13:58:56.506020 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.503452 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 06 13:58:56.509729 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.509429 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 13:58:56.509729 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.509521 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 06 13:58:56.609009 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.608721 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609009 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.608770 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609009 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.608808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609009 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.608835 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52227af2-b42c-4796-ac5a-43dca42cd06d-config-out\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609009 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.608881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609649 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.609133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609649 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.609172 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609649 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.609204 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609649 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.609232 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4krxg\" (UniqueName: \"kubernetes.io/projected/52227af2-b42c-4796-ac5a-43dca42cd06d-kube-api-access-4krxg\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609649 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.609275 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52227af2-b42c-4796-ac5a-43dca42cd06d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609649 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.609307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609649 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.609389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-web-config\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609649 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.609425 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609649 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.609449 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609649 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.609477 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609649 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.609500 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609649 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.609533 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-config\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609649 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.609567 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609649 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.609591 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.609649 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.609621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.711345 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.711137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712475 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-config\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712504 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52227af2-b42c-4796-ac5a-43dca42cd06d-config-out\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712698 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712754 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712784 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4krxg\" (UniqueName: \"kubernetes.io/projected/52227af2-b42c-4796-ac5a-43dca42cd06d-kube-api-access-4krxg\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.713137 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712862 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52227af2-b42c-4796-ac5a-43dca42cd06d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.714055 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.714055 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.712990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-web-config\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.715001 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.714460 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.715001 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.714817 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.715990 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.715773 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.716685 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.716256 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.716954 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.716931 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.718417 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.717481 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.718417 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.718074 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.719285 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.718750 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-web-config\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.719285 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.719200 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.720568 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.720501 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52227af2-b42c-4796-ac5a-43dca42cd06d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.722124 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.722102 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.722552 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.722528 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52227af2-b42c-4796-ac5a-43dca42cd06d-config-out\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.723408 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.723363 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.725617 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.725555 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.726536 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.726458 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.727057 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.727016 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4krxg\" (UniqueName: \"kubernetes.io/projected/52227af2-b42c-4796-ac5a-43dca42cd06d-kube-api-access-4krxg\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.727194 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.727105 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.727429 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.727405 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-config\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.727588 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.727565 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.728098 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.728064 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.821657 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:56.820959 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:58:56.874035 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:56.873987 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4f0419_be6f_4900_8116_6fdbe7278997.slice/crio-eac5cb69f76245a416fa8bd04a04c36ea54ebe31cdda9b8da547a7dff4a1c42c WatchSource:0}: Error finding container eac5cb69f76245a416fa8bd04a04c36ea54ebe31cdda9b8da547a7dff4a1c42c: Status 404 returned error can't find the container with id eac5cb69f76245a416fa8bd04a04c36ea54ebe31cdda9b8da547a7dff4a1c42c Sep 06 13:58:57.233006 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:57.232143 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 13:58:57.237229 ip-10-0-132-113 kubenswrapper[2577]: W0906 13:58:57.237187 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52227af2_b42c_4796_ac5a_43dca42cd06d.slice/crio-254ac562be9167cc79ce71b2e1290c9d9d2280cedc1993e8a1c8e8cbe7dfffe6 WatchSource:0}: Error finding container 254ac562be9167cc79ce71b2e1290c9d9d2280cedc1993e8a1c8e8cbe7dfffe6: Status 404 returned error can't find the container with id 254ac562be9167cc79ce71b2e1290c9d9d2280cedc1993e8a1c8e8cbe7dfffe6 Sep 06 13:58:57.650293 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:57.650044 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bkxp5" event={"ID":"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd","Type":"ContainerStarted","Data":"01135510a3b7759a20ee965ce00ecdb48ff6e1ccad27b5279634ec410475e752"} Sep 06 13:58:57.650293 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:57.650113 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bkxp5" event={"ID":"a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd","Type":"ContainerStarted","Data":"ba34f4fe749ea297a4e99d01d1d55a4794a68f89ec669feb6b61113b3183aa64"} Sep 06 13:58:57.654210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:57.653118 2577 generic.go:356] "Generic (PLEG): container finished" podID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerID="e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17" exitCode=0 Sep 06 13:58:57.654210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:57.653190 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52227af2-b42c-4796-ac5a-43dca42cd06d","Type":"ContainerDied","Data":"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17"} Sep 06 13:58:57.654210 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:57.653253 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52227af2-b42c-4796-ac5a-43dca42cd06d","Type":"ContainerStarted","Data":"254ac562be9167cc79ce71b2e1290c9d9d2280cedc1993e8a1c8e8cbe7dfffe6"} Sep 06 13:58:57.656868 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:57.656833 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5" event={"ID":"3b4f0419-be6f-4900-8116-6fdbe7278997","Type":"ContainerStarted","Data":"eac5cb69f76245a416fa8bd04a04c36ea54ebe31cdda9b8da547a7dff4a1c42c"} Sep 06 13:58:57.661313 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:57.661289 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" event={"ID":"9eb9bded-0cf1-41ab-b338-87538c1644a6","Type":"ContainerStarted","Data":"52d92595e95f6508e42ed4f3f6316fb3535e59ae7fdef9a7d93dac9000da24ff"} Sep 06 13:58:57.661415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:57.661360 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" event={"ID":"9eb9bded-0cf1-41ab-b338-87538c1644a6","Type":"ContainerStarted","Data":"76634b40d83d0af7461889e609cf8e9d210dd9cba167445837afe3965c8c92ec"} Sep 06 13:58:57.661415 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:57.661404 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" event={"ID":"9eb9bded-0cf1-41ab-b338-87538c1644a6","Type":"ContainerStarted","Data":"f5e6c9045b2f2f0ae2745b9ad4b7556d400c5cf813fcb3954645a1054827998c"} Sep 06 13:58:57.668205 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:57.667764 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bkxp5" podStartSLOduration=66.221842917 podStartE2EDuration="1m8.667746171s" totalImagesPullingTime="2.445903254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:57.665576945 +0000 UTC m=+69.121065808" watchObservedRunningTime="2026-09-06 13:58:57.667746171 +0000 UTC m=+69.123235036" Sep 06 13:58:57.754263 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:57.754222 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:58:57.806683 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:58:57.806647 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:00.609606 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:00.609565 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:00.707662 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:00.707592 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:00.758632 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:00.758595 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:00.866093 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:00.866016 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:00.916250 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:00.916212 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:02.037850 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:02.037812 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:02.134739 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:02.134705 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:04.527624 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:04.527582 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:04.574746 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:04.574714 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:05.014514 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:05.014435 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:05.115337 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:05.115296 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:05.702464 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:05.702430 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:05.755281 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:05.755244 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:07.726035 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:07.725996 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:07.779513 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:07.779461 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:09.142034 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:09.142001 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:09.229821 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:09.229786 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:11.729931 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.729875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9fqd2" event={"ID":"53540329-a02b-4fcb-ac80-8d6bf3489b10","Type":"ContainerStarted","Data":"52f4b5d6c4fa417449fca0a51f394d72269ea82687a0f5f2de036e43bb140682"} Sep 06 13:59:11.738923 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.738748 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52227af2-b42c-4796-ac5a-43dca42cd06d","Type":"ContainerStarted","Data":"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925"} Sep 06 13:59:11.738923 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.738820 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52227af2-b42c-4796-ac5a-43dca42cd06d","Type":"ContainerStarted","Data":"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1"} Sep 06 13:59:11.738923 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.738858 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52227af2-b42c-4796-ac5a-43dca42cd06d","Type":"ContainerStarted","Data":"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857"} Sep 06 13:59:11.738923 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.738894 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52227af2-b42c-4796-ac5a-43dca42cd06d","Type":"ContainerStarted","Data":"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e"} Sep 06 13:59:11.745288 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.744110 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5" event={"ID":"3b4f0419-be6f-4900-8116-6fdbe7278997","Type":"ContainerStarted","Data":"61008fba7607426aae3a9578bc6ead8e809dcc9322e2a1cfd57eff4a4a288967"} Sep 06 13:59:11.745288 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.745234 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5" Sep 06 13:59:11.750738 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.750438 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerStarted","Data":"3d28b05572347954a24dad38b7fb4c6dfdc209a9e196ca1b802ace841d6b001d"} Sep 06 13:59:11.750738 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.750504 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerStarted","Data":"7b6a39b0d3b5e485da69cd6f1e44df7b9b74ca37d9395c6f72d76ca015dd44ed"} Sep 06 13:59:11.750738 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.750541 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerStarted","Data":"39df093bf2798475e4d7e4b287a17c20e304bd02a960d018aa18c1a963f34fad"} Sep 06 13:59:11.750738 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.750577 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerStarted","Data":"2a368b5ed27585cf00864d22eddb925a5dcde84401d001411019cc4a37ac84f2"} Sep 06 13:59:11.755759 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.754077 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9fqd2" podStartSLOduration=65.090132649 podStartE2EDuration="1m21.754058559s" totalImagesPullingTime="16.66392591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:59:11.753414235 +0000 UTC m=+83.208903097" watchObservedRunningTime="2026-09-06 13:59:11.754058559 +0000 UTC m=+83.209547423" Sep 06 13:59:11.756502 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.756401 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5" Sep 06 13:59:11.757441 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.756773 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" event={"ID":"e080444a-1143-4e22-bcbe-b2180243a549","Type":"ContainerStarted","Data":"4c630197f5d4ef1c536295b9446c3d9a3ab145685e0f9f5b4aa505633b38e029"} Sep 06 13:59:11.762899 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.762867 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gkcs9" event={"ID":"83eae010-1fa5-4861-ad26-9be01d46a741","Type":"ContainerStarted","Data":"7410662d84e1343f12341b5798516d594323275eaf3823d1dddbf592016e231c"} Sep 06 13:59:11.764185 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.764170 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:59:11.769303 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.769105 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-4h9mc" event={"ID":"c7c35f62-5d92-4fd0-82b4-cc6274ecd0c1","Type":"ContainerStarted","Data":"34458ed278fa1f778681a72349ad6cda67b64ab4ba5cc0e77ae08421af957c9b"} Sep 06 13:59:11.769303 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.769248 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-4h9mc" Sep 06 13:59:11.771947 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.771505 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjbv5" podStartSLOduration=3.543811195 podStartE2EDuration="17.771488114s" totalImagesPullingTime="14.227676919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:59:11.771009467 +0000 UTC m=+83.226498335" watchObservedRunningTime="2026-09-06 13:59:11.771488114 +0000 UTC m=+83.226976978" Sep 06 13:59:11.776627 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.774527 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" event={"ID":"9eb9bded-0cf1-41ab-b338-87538c1644a6","Type":"ContainerStarted","Data":"0a5657cf02bfa974cfeb064296458e5de5ade1b1db48cd640b785f4ef1cd7d67"} Sep 06 13:59:11.776627 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.774605 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" event={"ID":"9eb9bded-0cf1-41ab-b338-87538c1644a6","Type":"ContainerStarted","Data":"251094ca9f9a9b64ed4a579e5afd0b90888afdd7dd08a117dcd42f3247c6e6da"} Sep 06 13:59:11.776627 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.774651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" event={"ID":"9eb9bded-0cf1-41ab-b338-87538c1644a6","Type":"ContainerStarted","Data":"59a7d09cce48538dfd176381a2210e29cd2da3a92f4ee73ee51d5eba58f56cab"} Sep 06 13:59:11.776627 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.775582 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:59:11.779402 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.779385 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-4h9mc" Sep 06 13:59:11.785173 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.784344 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" Sep 06 13:59:11.819037 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.812270 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-4h9mc" podStartSLOduration=1.403089602 podStartE2EDuration="22.812234322s" totalImagesPullingTime="21.40914472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:59:11.788363881 +0000 UTC m=+83.243852745" watchObservedRunningTime="2026-09-06 13:59:11.812234322 +0000 UTC m=+83.267723186" Sep 06 13:59:11.847627 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.847512 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gkcs9" podStartSLOduration=66.46758833 podStartE2EDuration="1m22.847491265s" totalImagesPullingTime="16.379902935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:59:11.829861996 +0000 UTC m=+83.285350859" watchObservedRunningTime="2026-09-06 13:59:11.847491265 +0000 UTC m=+83.302980135" Sep 06 13:59:11.848100 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.847873 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" podStartSLOduration=1.9902663600000001 podStartE2EDuration="17.847864331s" totalImagesPullingTime="15.857597971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:59:11.845887165 +0000 UTC m=+83.301376040" watchObservedRunningTime="2026-09-06 13:59:11.847864331 +0000 UTC m=+83.303353194" Sep 06 13:59:11.867638 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:11.867482 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7876d5bf99-mn6rr" podStartSLOduration=2.795740762 podStartE2EDuration="19.867462435s" totalImagesPullingTime="17.071721673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:59:11.866556028 +0000 UTC m=+83.322044892" watchObservedRunningTime="2026-09-06 13:59:11.867462435 +0000 UTC m=+83.322951298" Sep 06 13:59:12.786625 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:12.786578 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52227af2-b42c-4796-ac5a-43dca42cd06d","Type":"ContainerStarted","Data":"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3"} Sep 06 13:59:12.787556 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:12.787531 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52227af2-b42c-4796-ac5a-43dca42cd06d","Type":"ContainerStarted","Data":"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2"} Sep 06 13:59:12.792260 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:12.792228 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerStarted","Data":"eea5edf8236e576a1c954c665d54cb04d6ac6580879506808b3fa0c14503be88"} Sep 06 13:59:12.792367 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:12.792324 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerStarted","Data":"ce7fc2cc73d40f5e41457591380a474d06477c9dab06198db2664915348119c2"} Sep 06 13:59:12.812591 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:12.812534 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.233222986 podStartE2EDuration="16.812517374s" totalImagesPullingTime="13.53937687s" totalInitContainerRuntime="39.917518ms" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:59:12.81220109 +0000 UTC m=+84.267689953" watchObservedRunningTime="2026-09-06 13:59:12.812517374 +0000 UTC m=+84.268006249" Sep 06 13:59:12.843010 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:12.842948 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.035682594 podStartE2EDuration="21.842892617s" totalImagesPullingTime="16.784142241s" totalInitContainerRuntime="23.067782ms" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:59:12.841899115 +0000 UTC m=+84.297387979" watchObservedRunningTime="2026-09-06 13:59:12.842892617 +0000 UTC m=+84.298381478" Sep 06 13:59:14.337240 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:14.337187 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:14.386571 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:14.386542 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:14.777015 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:14.776979 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:14.877638 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:14.877550 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:14.963386 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:14.963355 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:59:14.963567 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:14.963437 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:59:15.958211 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:15.958174 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:16.009881 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:16.009844 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:16.823012 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:16.822976 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:59:19.958113 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:19.958072 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:20.058218 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:20.058179 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:20.105204 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:20.105175 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:23.366995 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:23.366955 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:23.466731 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:23.466699 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:34.921251 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:34.921214 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:34.968412 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:34.968387 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:59:34.972113 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:34.972095 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6996bf4777-kh87z" Sep 06 13:59:34.975755 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:34.975733 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:38.374726 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:38.374692 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:38.426815 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:38.426780 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:43.800456 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:43.800424 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gkcs9" Sep 06 13:59:53.080921 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:53.080871 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 13:59:56.822457 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:56.822421 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:59:56.884075 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:56.884037 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 13:59:56.996486 ip-10-0-132-113 kubenswrapper[2577]: I0906 13:59:56.996452 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:09.404365 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:09.404325 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:00:09.983326 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:09.983294 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:00:10.484320 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:10.484290 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 14:00:10.484669 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:10.484584 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="alertmanager" containerID="cri-o://2a368b5ed27585cf00864d22eddb925a5dcde84401d001411019cc4a37ac84f2" gracePeriod=120 Sep 06 14:00:10.484669 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:10.484649 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="kube-rbac-proxy-metric" containerID="cri-o://ce7fc2cc73d40f5e41457591380a474d06477c9dab06198db2664915348119c2" gracePeriod=120 Sep 06 14:00:10.484766 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:10.484673 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="kube-rbac-proxy-web" containerID="cri-o://7b6a39b0d3b5e485da69cd6f1e44df7b9b74ca37d9395c6f72d76ca015dd44ed" gracePeriod=120 Sep 06 14:00:10.484766 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:10.484705 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="config-reloader" containerID="cri-o://39df093bf2798475e4d7e4b287a17c20e304bd02a960d018aa18c1a963f34fad" gracePeriod=120 Sep 06 14:00:10.484864 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:10.484749 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="kube-rbac-proxy" containerID="cri-o://3d28b05572347954a24dad38b7fb4c6dfdc209a9e196ca1b802ace841d6b001d" gracePeriod=120 Sep 06 14:00:10.484864 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:10.484744 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="prom-label-proxy" containerID="cri-o://eea5edf8236e576a1c954c665d54cb04d6ac6580879506808b3fa0c14503be88" gracePeriod=120 Sep 06 14:00:11.039552 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.039520 2577 generic.go:356] "Generic (PLEG): container finished" podID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerID="eea5edf8236e576a1c954c665d54cb04d6ac6580879506808b3fa0c14503be88" exitCode=0 Sep 06 14:00:11.039552 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.039543 2577 generic.go:356] "Generic (PLEG): container finished" podID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerID="ce7fc2cc73d40f5e41457591380a474d06477c9dab06198db2664915348119c2" exitCode=0 Sep 06 14:00:11.039552 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.039550 2577 generic.go:356] "Generic (PLEG): container finished" podID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerID="3d28b05572347954a24dad38b7fb4c6dfdc209a9e196ca1b802ace841d6b001d" exitCode=0 Sep 06 14:00:11.039552 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.039556 2577 generic.go:356] "Generic (PLEG): container finished" podID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerID="39df093bf2798475e4d7e4b287a17c20e304bd02a960d018aa18c1a963f34fad" exitCode=0 Sep 06 14:00:11.039552 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.039561 2577 generic.go:356] "Generic (PLEG): container finished" podID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerID="2a368b5ed27585cf00864d22eddb925a5dcde84401d001411019cc4a37ac84f2" exitCode=0 Sep 06 14:00:11.039826 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.039589 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerDied","Data":"eea5edf8236e576a1c954c665d54cb04d6ac6580879506808b3fa0c14503be88"} Sep 06 14:00:11.039826 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.039651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerDied","Data":"ce7fc2cc73d40f5e41457591380a474d06477c9dab06198db2664915348119c2"} Sep 06 14:00:11.039826 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.039679 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerDied","Data":"3d28b05572347954a24dad38b7fb4c6dfdc209a9e196ca1b802ace841d6b001d"} Sep 06 14:00:11.039826 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.039706 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerDied","Data":"39df093bf2798475e4d7e4b287a17c20e304bd02a960d018aa18c1a963f34fad"} Sep 06 14:00:11.039826 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.039728 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerDied","Data":"2a368b5ed27585cf00864d22eddb925a5dcde84401d001411019cc4a37ac84f2"} Sep 06 14:00:11.725016 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.724992 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:11.774009 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.773967 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-config-out\") pod \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " Sep 06 14:00:11.774009 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.774005 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-tls-assets\") pod \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " Sep 06 14:00:11.774247 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.774046 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-alertmanager-trusted-ca-bundle\") pod \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " Sep 06 14:00:11.774247 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.774069 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-metrics-client-ca\") pod \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " Sep 06 14:00:11.774247 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.774111 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-config-volume\") pod \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " Sep 06 14:00:11.774247 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.774139 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-web-config\") pod \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " Sep 06 14:00:11.774247 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.774163 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy\") pod \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " Sep 06 14:00:11.774247 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.774185 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " Sep 06 14:00:11.774247 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.774216 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccq8z\" (UniqueName: \"kubernetes.io/projected/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-kube-api-access-ccq8z\") pod \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " Sep 06 14:00:11.774573 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.774254 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-alertmanager-main-db\") pod \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " Sep 06 14:00:11.774573 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.774304 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy-web\") pod \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " Sep 06 14:00:11.774573 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.774361 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-cluster-tls-config\") pod \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " Sep 06 14:00:11.774573 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.774392 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-main-tls\") pod \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\" (UID: \"3ba37183-e831-4d29-82bd-9c9cb9a7ba52\") " Sep 06 14:00:11.776138 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.775418 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "3ba37183-e831-4d29-82bd-9c9cb9a7ba52" (UID: "3ba37183-e831-4d29-82bd-9c9cb9a7ba52"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:00:11.776138 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.775835 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "3ba37183-e831-4d29-82bd-9c9cb9a7ba52" (UID: "3ba37183-e831-4d29-82bd-9c9cb9a7ba52"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:00:11.777529 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.777495 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "3ba37183-e831-4d29-82bd-9c9cb9a7ba52" (UID: "3ba37183-e831-4d29-82bd-9c9cb9a7ba52"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:00:11.777669 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.777645 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "3ba37183-e831-4d29-82bd-9c9cb9a7ba52" (UID: "3ba37183-e831-4d29-82bd-9c9cb9a7ba52"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:11.777786 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.777756 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "3ba37183-e831-4d29-82bd-9c9cb9a7ba52" (UID: "3ba37183-e831-4d29-82bd-9c9cb9a7ba52"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:11.778220 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.777868 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3ba37183-e831-4d29-82bd-9c9cb9a7ba52" (UID: "3ba37183-e831-4d29-82bd-9c9cb9a7ba52"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:00:11.778220 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.777935 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-config-out" (OuterVolumeSpecName: "config-out") pod "3ba37183-e831-4d29-82bd-9c9cb9a7ba52" (UID: "3ba37183-e831-4d29-82bd-9c9cb9a7ba52"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:00:11.779633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.779579 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "3ba37183-e831-4d29-82bd-9c9cb9a7ba52" (UID: "3ba37183-e831-4d29-82bd-9c9cb9a7ba52"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:11.779896 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.779866 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-kube-api-access-ccq8z" (OuterVolumeSpecName: "kube-api-access-ccq8z") pod "3ba37183-e831-4d29-82bd-9c9cb9a7ba52" (UID: "3ba37183-e831-4d29-82bd-9c9cb9a7ba52"). InnerVolumeSpecName "kube-api-access-ccq8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:00:11.780004 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.779886 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-config-volume" (OuterVolumeSpecName: "config-volume") pod "3ba37183-e831-4d29-82bd-9c9cb9a7ba52" (UID: "3ba37183-e831-4d29-82bd-9c9cb9a7ba52"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:11.781025 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.780998 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "3ba37183-e831-4d29-82bd-9c9cb9a7ba52" (UID: "3ba37183-e831-4d29-82bd-9c9cb9a7ba52"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:11.784995 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.784960 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "3ba37183-e831-4d29-82bd-9c9cb9a7ba52" (UID: "3ba37183-e831-4d29-82bd-9c9cb9a7ba52"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:11.790566 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.790542 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-web-config" (OuterVolumeSpecName: "web-config") pod "3ba37183-e831-4d29-82bd-9c9cb9a7ba52" (UID: "3ba37183-e831-4d29-82bd-9c9cb9a7ba52"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:11.874974 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.874944 2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:11.875123 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.874987 2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-web-config\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:11.875123 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.875000 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:11.875123 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.875009 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:11.875123 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.875018 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccq8z\" (UniqueName: \"kubernetes.io/projected/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-kube-api-access-ccq8z\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:11.875123 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.875027 2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-alertmanager-main-db\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:11.875123 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.875036 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:11.875123 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.875044 2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-cluster-tls-config\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:11.875123 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.875052 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-secret-alertmanager-main-tls\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:11.875123 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.875061 2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-config-out\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:11.875123 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.875069 2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-tls-assets\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:11.875123 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.875077 2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:11.875123 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:11.875084 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ba37183-e831-4d29-82bd-9c9cb9a7ba52-metrics-client-ca\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:12.045788 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.045708 2577 generic.go:356] "Generic (PLEG): container finished" podID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerID="7b6a39b0d3b5e485da69cd6f1e44df7b9b74ca37d9395c6f72d76ca015dd44ed" exitCode=0 Sep 06 14:00:12.045943 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.045791 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.045943 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.045791 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerDied","Data":"7b6a39b0d3b5e485da69cd6f1e44df7b9b74ca37d9395c6f72d76ca015dd44ed"} Sep 06 14:00:12.045943 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.045926 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ba37183-e831-4d29-82bd-9c9cb9a7ba52","Type":"ContainerDied","Data":"30a56a62b2c817d944ae709cd048f27cae6da7b94002bd54eae0020fe64e398b"} Sep 06 14:00:12.046049 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.045929 2577 scope.go:117] "RemoveContainer" containerID="eea5edf8236e576a1c954c665d54cb04d6ac6580879506808b3fa0c14503be88" Sep 06 14:00:12.053818 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.053796 2577 scope.go:117] "RemoveContainer" containerID="ce7fc2cc73d40f5e41457591380a474d06477c9dab06198db2664915348119c2" Sep 06 14:00:12.060720 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.060703 2577 scope.go:117] "RemoveContainer" containerID="3d28b05572347954a24dad38b7fb4c6dfdc209a9e196ca1b802ace841d6b001d" Sep 06 14:00:12.068715 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.068678 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 14:00:12.069434 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.069409 2577 scope.go:117] "RemoveContainer" containerID="7b6a39b0d3b5e485da69cd6f1e44df7b9b74ca37d9395c6f72d76ca015dd44ed" Sep 06 14:00:12.072515 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.072493 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 14:00:12.076741 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.076719 2577 scope.go:117] "RemoveContainer" containerID="39df093bf2798475e4d7e4b287a17c20e304bd02a960d018aa18c1a963f34fad" Sep 06 14:00:12.083601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.083582 2577 scope.go:117] "RemoveContainer" containerID="2a368b5ed27585cf00864d22eddb925a5dcde84401d001411019cc4a37ac84f2" Sep 06 14:00:12.090645 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.090628 2577 scope.go:117] "RemoveContainer" containerID="5fe0fe7f5346bec0d564615124b4d3e3df3889288a5dda81bfda1884581ebc16" Sep 06 14:00:12.097398 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.097296 2577 scope.go:117] "RemoveContainer" containerID="eea5edf8236e576a1c954c665d54cb04d6ac6580879506808b3fa0c14503be88" Sep 06 14:00:12.097666 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:00:12.097616 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea5edf8236e576a1c954c665d54cb04d6ac6580879506808b3fa0c14503be88\": container with ID starting with eea5edf8236e576a1c954c665d54cb04d6ac6580879506808b3fa0c14503be88 not found: ID does not exist" containerID="eea5edf8236e576a1c954c665d54cb04d6ac6580879506808b3fa0c14503be88" Sep 06 14:00:12.097750 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.097657 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea5edf8236e576a1c954c665d54cb04d6ac6580879506808b3fa0c14503be88"} err="failed to get container status \"eea5edf8236e576a1c954c665d54cb04d6ac6580879506808b3fa0c14503be88\": rpc error: code = NotFound desc = could not find container \"eea5edf8236e576a1c954c665d54cb04d6ac6580879506808b3fa0c14503be88\": container with ID starting with eea5edf8236e576a1c954c665d54cb04d6ac6580879506808b3fa0c14503be88 not found: ID does not exist" Sep 06 14:00:12.097750 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.097708 2577 scope.go:117] "RemoveContainer" containerID="ce7fc2cc73d40f5e41457591380a474d06477c9dab06198db2664915348119c2" Sep 06 14:00:12.098397 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:00:12.098022 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce7fc2cc73d40f5e41457591380a474d06477c9dab06198db2664915348119c2\": container with ID starting with ce7fc2cc73d40f5e41457591380a474d06477c9dab06198db2664915348119c2 not found: ID does not exist" containerID="ce7fc2cc73d40f5e41457591380a474d06477c9dab06198db2664915348119c2" Sep 06 14:00:12.098397 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.098054 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce7fc2cc73d40f5e41457591380a474d06477c9dab06198db2664915348119c2"} err="failed to get container status \"ce7fc2cc73d40f5e41457591380a474d06477c9dab06198db2664915348119c2\": rpc error: code = NotFound desc = could not find container \"ce7fc2cc73d40f5e41457591380a474d06477c9dab06198db2664915348119c2\": container with ID starting with ce7fc2cc73d40f5e41457591380a474d06477c9dab06198db2664915348119c2 not found: ID does not exist" Sep 06 14:00:12.098397 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.098073 2577 scope.go:117] "RemoveContainer" containerID="3d28b05572347954a24dad38b7fb4c6dfdc209a9e196ca1b802ace841d6b001d" Sep 06 14:00:12.098603 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:00:12.098392 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d28b05572347954a24dad38b7fb4c6dfdc209a9e196ca1b802ace841d6b001d\": container with ID starting with 3d28b05572347954a24dad38b7fb4c6dfdc209a9e196ca1b802ace841d6b001d not found: ID does not exist" containerID="3d28b05572347954a24dad38b7fb4c6dfdc209a9e196ca1b802ace841d6b001d" Sep 06 14:00:12.098603 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.098419 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d28b05572347954a24dad38b7fb4c6dfdc209a9e196ca1b802ace841d6b001d"} err="failed to get container status \"3d28b05572347954a24dad38b7fb4c6dfdc209a9e196ca1b802ace841d6b001d\": rpc error: code = NotFound desc = could not find container \"3d28b05572347954a24dad38b7fb4c6dfdc209a9e196ca1b802ace841d6b001d\": container with ID starting with 3d28b05572347954a24dad38b7fb4c6dfdc209a9e196ca1b802ace841d6b001d not found: ID does not exist" Sep 06 14:00:12.098603 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.098439 2577 scope.go:117] "RemoveContainer" containerID="7b6a39b0d3b5e485da69cd6f1e44df7b9b74ca37d9395c6f72d76ca015dd44ed" Sep 06 14:00:12.099061 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:00:12.099037 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6a39b0d3b5e485da69cd6f1e44df7b9b74ca37d9395c6f72d76ca015dd44ed\": container with ID starting with 7b6a39b0d3b5e485da69cd6f1e44df7b9b74ca37d9395c6f72d76ca015dd44ed not found: ID does not exist" containerID="7b6a39b0d3b5e485da69cd6f1e44df7b9b74ca37d9395c6f72d76ca015dd44ed" Sep 06 14:00:12.099173 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.099065 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6a39b0d3b5e485da69cd6f1e44df7b9b74ca37d9395c6f72d76ca015dd44ed"} err="failed to get container status \"7b6a39b0d3b5e485da69cd6f1e44df7b9b74ca37d9395c6f72d76ca015dd44ed\": rpc error: code = NotFound desc = could not find container \"7b6a39b0d3b5e485da69cd6f1e44df7b9b74ca37d9395c6f72d76ca015dd44ed\": container with ID starting with 7b6a39b0d3b5e485da69cd6f1e44df7b9b74ca37d9395c6f72d76ca015dd44ed not found: ID does not exist" Sep 06 14:00:12.099173 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.099085 2577 scope.go:117] "RemoveContainer" containerID="39df093bf2798475e4d7e4b287a17c20e304bd02a960d018aa18c1a963f34fad" Sep 06 14:00:12.099409 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:00:12.099383 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39df093bf2798475e4d7e4b287a17c20e304bd02a960d018aa18c1a963f34fad\": container with ID starting with 39df093bf2798475e4d7e4b287a17c20e304bd02a960d018aa18c1a963f34fad not found: ID does not exist" containerID="39df093bf2798475e4d7e4b287a17c20e304bd02a960d018aa18c1a963f34fad" Sep 06 14:00:12.099495 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.099413 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39df093bf2798475e4d7e4b287a17c20e304bd02a960d018aa18c1a963f34fad"} err="failed to get container status \"39df093bf2798475e4d7e4b287a17c20e304bd02a960d018aa18c1a963f34fad\": rpc error: code = NotFound desc = could not find container \"39df093bf2798475e4d7e4b287a17c20e304bd02a960d018aa18c1a963f34fad\": container with ID starting with 39df093bf2798475e4d7e4b287a17c20e304bd02a960d018aa18c1a963f34fad not found: ID does not exist" Sep 06 14:00:12.099495 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.099435 2577 scope.go:117] "RemoveContainer" containerID="2a368b5ed27585cf00864d22eddb925a5dcde84401d001411019cc4a37ac84f2" Sep 06 14:00:12.099748 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:00:12.099724 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a368b5ed27585cf00864d22eddb925a5dcde84401d001411019cc4a37ac84f2\": container with ID starting with 2a368b5ed27585cf00864d22eddb925a5dcde84401d001411019cc4a37ac84f2 not found: ID does not exist" containerID="2a368b5ed27585cf00864d22eddb925a5dcde84401d001411019cc4a37ac84f2" Sep 06 14:00:12.099828 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.099752 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a368b5ed27585cf00864d22eddb925a5dcde84401d001411019cc4a37ac84f2"} err="failed to get container status \"2a368b5ed27585cf00864d22eddb925a5dcde84401d001411019cc4a37ac84f2\": rpc error: code = NotFound desc = could not find container \"2a368b5ed27585cf00864d22eddb925a5dcde84401d001411019cc4a37ac84f2\": container with ID starting with 2a368b5ed27585cf00864d22eddb925a5dcde84401d001411019cc4a37ac84f2 not found: ID does not exist" Sep 06 14:00:12.099828 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.099772 2577 scope.go:117] "RemoveContainer" containerID="5fe0fe7f5346bec0d564615124b4d3e3df3889288a5dda81bfda1884581ebc16" Sep 06 14:00:12.099963 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.099878 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 14:00:12.100101 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:00:12.100080 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe0fe7f5346bec0d564615124b4d3e3df3889288a5dda81bfda1884581ebc16\": container with ID starting with 5fe0fe7f5346bec0d564615124b4d3e3df3889288a5dda81bfda1884581ebc16 not found: ID does not exist" containerID="5fe0fe7f5346bec0d564615124b4d3e3df3889288a5dda81bfda1884581ebc16" Sep 06 14:00:12.100139 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100109 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe0fe7f5346bec0d564615124b4d3e3df3889288a5dda81bfda1884581ebc16"} err="failed to get container status \"5fe0fe7f5346bec0d564615124b4d3e3df3889288a5dda81bfda1884581ebc16\": rpc error: code = NotFound desc = could not find container \"5fe0fe7f5346bec0d564615124b4d3e3df3889288a5dda81bfda1884581ebc16\": container with ID starting with 5fe0fe7f5346bec0d564615124b4d3e3df3889288a5dda81bfda1884581ebc16 not found: ID does not exist" Sep 06 14:00:12.100248 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100236 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="config-reloader" Sep 06 14:00:12.100282 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100250 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="config-reloader" Sep 06 14:00:12.100282 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100260 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="init-config-reloader" Sep 06 14:00:12.100282 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100265 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="init-config-reloader" Sep 06 14:00:12.100282 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100272 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="kube-rbac-proxy" Sep 06 14:00:12.100282 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100278 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="kube-rbac-proxy" Sep 06 14:00:12.100419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100290 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="kube-rbac-proxy-web" Sep 06 14:00:12.100419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100295 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="kube-rbac-proxy-web" Sep 06 14:00:12.100419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100305 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="prom-label-proxy" Sep 06 14:00:12.100419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100310 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="prom-label-proxy" Sep 06 14:00:12.100419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100318 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="kube-rbac-proxy-metric" Sep 06 14:00:12.100419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100323 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="kube-rbac-proxy-metric" Sep 06 14:00:12.100419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100329 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="alertmanager" Sep 06 14:00:12.100419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100334 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="alertmanager" Sep 06 14:00:12.100419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100384 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="kube-rbac-proxy-web" Sep 06 14:00:12.100419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100393 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="config-reloader" Sep 06 14:00:12.100419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100399 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="kube-rbac-proxy-metric" Sep 06 14:00:12.100419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100405 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="alertmanager" Sep 06 14:00:12.100419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100410 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="kube-rbac-proxy" Sep 06 14:00:12.100419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.100417 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" containerName="prom-label-proxy" Sep 06 14:00:12.103219 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.103201 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.105468 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.105445 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 06 14:00:12.105574 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.105528 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 06 14:00:12.105643 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.105580 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 06 14:00:12.105643 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.105609 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xknm6\"" Sep 06 14:00:12.105888 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.105867 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 06 14:00:12.105888 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.105879 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 06 14:00:12.106047 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.105873 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 06 14:00:12.106047 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.105922 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 06 14:00:12.106149 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.106134 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 06 14:00:12.111368 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.111345 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 06 14:00:12.119717 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.119696 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 14:00:12.177744 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.177699 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.177884 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.177747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.177884 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.177815 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-web-config\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.177884 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.177850 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f230577-b8d8-4b16-92e3-a6b51927611f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.177884 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.177876 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f230577-b8d8-4b16-92e3-a6b51927611f-config-out\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.178067 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.177898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.178067 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.177951 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.178067 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.177982 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.178067 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.178015 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-config-volume\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.178067 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.178036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f230577-b8d8-4b16-92e3-a6b51927611f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.178067 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.178060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f230577-b8d8-4b16-92e3-a6b51927611f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.178289 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.178075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f230577-b8d8-4b16-92e3-a6b51927611f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.178289 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.178138 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjskn\" (UniqueName: \"kubernetes.io/projected/9f230577-b8d8-4b16-92e3-a6b51927611f-kube-api-access-jjskn\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.278678 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.278640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.278678 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.278682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-config-volume\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.278931 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.278712 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f230577-b8d8-4b16-92e3-a6b51927611f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.278931 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.278744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f230577-b8d8-4b16-92e3-a6b51927611f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.278931 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.278765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f230577-b8d8-4b16-92e3-a6b51927611f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.278931 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.278791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjskn\" (UniqueName: \"kubernetes.io/projected/9f230577-b8d8-4b16-92e3-a6b51927611f-kube-api-access-jjskn\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.278931 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.278831 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.278931 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.278865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.279235 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.278944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-web-config\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.279235 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.278974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f230577-b8d8-4b16-92e3-a6b51927611f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.279235 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.278997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f230577-b8d8-4b16-92e3-a6b51927611f-config-out\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.279235 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.279038 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.279235 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.279082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.279483 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.279305 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f230577-b8d8-4b16-92e3-a6b51927611f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.279940 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.279632 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f230577-b8d8-4b16-92e3-a6b51927611f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.281945 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.281923 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.282066 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.281955 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f230577-b8d8-4b16-92e3-a6b51927611f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.282392 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.282348 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f230577-b8d8-4b16-92e3-a6b51927611f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.282392 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.282358 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f230577-b8d8-4b16-92e3-a6b51927611f-config-out\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.282392 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.282382 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-config-volume\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.282548 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.282455 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-web-config\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.282548 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.282500 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.282621 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.282563 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.282832 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.282811 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.283520 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.283505 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f230577-b8d8-4b16-92e3-a6b51927611f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.286482 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.286465 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjskn\" (UniqueName: \"kubernetes.io/projected/9f230577-b8d8-4b16-92e3-a6b51927611f-kube-api-access-jjskn\") pod \"alertmanager-main-0\" (UID: \"9f230577-b8d8-4b16-92e3-a6b51927611f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.413011 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.412930 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 06 14:00:12.686004 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:12.685977 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 14:00:12.687967 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:00:12.687939 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f230577_b8d8_4b16_92e3_a6b51927611f.slice/crio-00216734aeb93f5eacb8668b62516f829982e1c97f95519c9d8d25eae873e2c3 WatchSource:0}: Error finding container 00216734aeb93f5eacb8668b62516f829982e1c97f95519c9d8d25eae873e2c3: Status 404 returned error can't find the container with id 00216734aeb93f5eacb8668b62516f829982e1c97f95519c9d8d25eae873e2c3 Sep 06 14:00:13.052185 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:13.052140 2577 generic.go:356] "Generic (PLEG): container finished" podID="9f230577-b8d8-4b16-92e3-a6b51927611f" containerID="b2040eb6f305ae434f1d35ee5526bc77b347cae9d8b9918d8b08e242832eb41e" exitCode=0 Sep 06 14:00:13.052532 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:13.052226 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f230577-b8d8-4b16-92e3-a6b51927611f","Type":"ContainerDied","Data":"b2040eb6f305ae434f1d35ee5526bc77b347cae9d8b9918d8b08e242832eb41e"} Sep 06 14:00:13.052532 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:13.052288 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f230577-b8d8-4b16-92e3-a6b51927611f","Type":"ContainerStarted","Data":"00216734aeb93f5eacb8668b62516f829982e1c97f95519c9d8d25eae873e2c3"} Sep 06 14:00:13.071808 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:13.071783 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ba37183-e831-4d29-82bd-9c9cb9a7ba52" path="/var/lib/kubelet/pods/3ba37183-e831-4d29-82bd-9c9cb9a7ba52/volumes" Sep 06 14:00:14.059618 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.059586 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f230577-b8d8-4b16-92e3-a6b51927611f","Type":"ContainerStarted","Data":"47c38ed6c25434b1266c61275ef4fd28b1f9110fe7b60ee0648e8a653a445a80"} Sep 06 14:00:14.060028 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.059647 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f230577-b8d8-4b16-92e3-a6b51927611f","Type":"ContainerStarted","Data":"372e07ab112f4574cc31dda5c100aec1080641395ad718d674a26cfb7bfd9454"} Sep 06 14:00:14.060028 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.059674 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f230577-b8d8-4b16-92e3-a6b51927611f","Type":"ContainerStarted","Data":"3f0d253498ea10398bb97fe44a5774d479c4218e6095703f3b73ba3ab1fb4c64"} Sep 06 14:00:14.060028 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.059700 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f230577-b8d8-4b16-92e3-a6b51927611f","Type":"ContainerStarted","Data":"6fbe3605e463104666250e764c121800b63695eafeb8498292ccf2930637c15c"} Sep 06 14:00:14.060028 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.059723 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f230577-b8d8-4b16-92e3-a6b51927611f","Type":"ContainerStarted","Data":"fec1d1ef296c140e8b1a1c2a924dd17a10c2ae3568de35e73ae6504ae9d8a211"} Sep 06 14:00:14.060028 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.059744 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f230577-b8d8-4b16-92e3-a6b51927611f","Type":"ContainerStarted","Data":"5ede15be50821437aa16abd6832e9415aa5b4503993ec9fad5b6ad503d7b6fad"} Sep 06 14:00:14.087552 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.087503 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.072375424 podStartE2EDuration="2.08749005s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.114626ms" isStatefulPod=false podCreationTimestamp="2026-09-06 14:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:00:14.084972045 +0000 UTC m=+145.540460908" watchObservedRunningTime="2026-09-06 14:00:14.08749005 +0000 UTC m=+145.542978934" Sep 06 14:00:14.478489 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.478404 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-567955b447-sm9q6"] Sep 06 14:00:14.481516 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.481492 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.484209 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.484180 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 06 14:00:14.484329 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.484207 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-99w5l\"" Sep 06 14:00:14.484743 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.484721 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 06 14:00:14.484876 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.484858 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 06 14:00:14.485067 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.485050 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 06 14:00:14.485212 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.485186 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 06 14:00:14.489326 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.489304 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 06 14:00:14.494049 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.494030 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-567955b447-sm9q6"] Sep 06 14:00:14.497681 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.497661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-federate-client-tls\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.497799 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.497694 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-telemeter-client-tls\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.497799 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.497722 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-serving-certs-ca-bundle\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.497799 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.497739 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-secret-telemeter-client\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.497799 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.497781 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.498093 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.497804 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.498093 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.497883 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lp6r\" (UniqueName: \"kubernetes.io/projected/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-kube-api-access-2lp6r\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.498093 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.497943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-metrics-client-ca\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.598666 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.598627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-serving-certs-ca-bundle\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.598666 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.598669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-secret-telemeter-client\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.598939 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.598688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.598939 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.598870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.598939 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.598916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lp6r\" (UniqueName: \"kubernetes.io/projected/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-kube-api-access-2lp6r\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.599106 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.598954 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-metrics-client-ca\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.599106 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.599009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-federate-client-tls\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.599106 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.599056 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-telemeter-client-tls\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.599464 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.599431 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-serving-certs-ca-bundle\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.599872 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.599820 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-metrics-client-ca\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.600007 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.599867 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.601529 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.601500 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-telemeter-client-tls\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.601529 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.601519 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-federate-client-tls\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.601690 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.601660 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.601750 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.601710 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-secret-telemeter-client\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.606234 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.606214 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lp6r\" (UniqueName: \"kubernetes.io/projected/c8b26446-fc0a-4422-b6fd-5771b7e9a14a-kube-api-access-2lp6r\") pod \"telemeter-client-567955b447-sm9q6\" (UID: \"c8b26446-fc0a-4422-b6fd-5771b7e9a14a\") " pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.751987 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.751507 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 14:00:14.751987 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.751881 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="prometheus" containerID="cri-o://686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e" gracePeriod=600 Sep 06 14:00:14.752482 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.752093 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="kube-rbac-proxy" containerID="cri-o://627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2" gracePeriod=600 Sep 06 14:00:14.752482 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.752145 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3" gracePeriod=600 Sep 06 14:00:14.752666 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.752160 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="thanos-sidecar" containerID="cri-o://81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1" gracePeriod=600 Sep 06 14:00:14.752666 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.752175 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="kube-rbac-proxy-web" containerID="cri-o://4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925" gracePeriod=600 Sep 06 14:00:14.752666 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.752189 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="config-reloader" containerID="cri-o://6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857" gracePeriod=600 Sep 06 14:00:14.793590 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.793562 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" Sep 06 14:00:14.999094 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:14.999063 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.071178 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.071134 2577 generic.go:356] "Generic (PLEG): container finished" podID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerID="a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3" exitCode=0 Sep 06 14:00:15.071178 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.071162 2577 generic.go:356] "Generic (PLEG): container finished" podID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerID="627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2" exitCode=0 Sep 06 14:00:15.071178 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.071170 2577 generic.go:356] "Generic (PLEG): container finished" podID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerID="4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925" exitCode=0 Sep 06 14:00:15.071178 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.071175 2577 generic.go:356] "Generic (PLEG): container finished" podID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerID="81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1" exitCode=0 Sep 06 14:00:15.071178 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.071181 2577 generic.go:356] "Generic (PLEG): container finished" podID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerID="6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857" exitCode=0 Sep 06 14:00:15.071178 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.071185 2577 generic.go:356] "Generic (PLEG): container finished" podID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerID="686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e" exitCode=0 Sep 06 14:00:15.071987 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.071225 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52227af2-b42c-4796-ac5a-43dca42cd06d","Type":"ContainerDied","Data":"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3"} Sep 06 14:00:15.071987 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.071410 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52227af2-b42c-4796-ac5a-43dca42cd06d","Type":"ContainerDied","Data":"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2"} Sep 06 14:00:15.071987 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.071408 2577 scope.go:117] "RemoveContainer" containerID="a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3" Sep 06 14:00:15.071987 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.071463 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52227af2-b42c-4796-ac5a-43dca42cd06d","Type":"ContainerDied","Data":"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925"} Sep 06 14:00:15.071987 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.071511 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52227af2-b42c-4796-ac5a-43dca42cd06d","Type":"ContainerDied","Data":"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1"} Sep 06 14:00:15.071987 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.071653 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52227af2-b42c-4796-ac5a-43dca42cd06d","Type":"ContainerDied","Data":"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857"} Sep 06 14:00:15.071987 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.071696 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52227af2-b42c-4796-ac5a-43dca42cd06d","Type":"ContainerDied","Data":"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e"} Sep 06 14:00:15.071987 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.071737 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52227af2-b42c-4796-ac5a-43dca42cd06d","Type":"ContainerDied","Data":"254ac562be9167cc79ce71b2e1290c9d9d2280cedc1993e8a1c8e8cbe7dfffe6"} Sep 06 14:00:15.071987 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.071772 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.075254 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.075230 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-567955b447-sm9q6"] Sep 06 14:00:15.078989 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:00:15.078967 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b26446_fc0a_4422_b6fd_5771b7e9a14a.slice/crio-50ab2101890623957316fffb47027fd655b287b6613351c78961081f20aa3210 WatchSource:0}: Error finding container 50ab2101890623957316fffb47027fd655b287b6613351c78961081f20aa3210: Status 404 returned error can't find the container with id 50ab2101890623957316fffb47027fd655b287b6613351c78961081f20aa3210 Sep 06 14:00:15.082332 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.082191 2577 scope.go:117] "RemoveContainer" containerID="627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2" Sep 06 14:00:15.090600 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.090582 2577 scope.go:117] "RemoveContainer" containerID="4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925" Sep 06 14:00:15.099452 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.099434 2577 scope.go:117] "RemoveContainer" containerID="81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1" Sep 06 14:00:15.103644 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.103623 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.103715 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.103655 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-1\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.103715 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.103690 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4krxg\" (UniqueName: \"kubernetes.io/projected/52227af2-b42c-4796-ac5a-43dca42cd06d-kube-api-access-4krxg\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.103797 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.103716 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-serving-certs-ca-bundle\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.103797 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.103739 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-grpc-tls\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.103797 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.103767 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-kube-rbac-proxy\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.103964 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.103802 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52227af2-b42c-4796-ac5a-43dca42cd06d-config-out\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.103964 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.103841 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52227af2-b42c-4796-ac5a-43dca42cd06d-tls-assets\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.103964 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.103871 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-web-config\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.103964 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.103896 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-config\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.103964 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.103957 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-trusted-ca-bundle\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.104196 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.103991 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-metrics-client-certs\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.104196 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.104017 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-kubelet-serving-ca-bundle\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.104196 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.104037 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:00:15.104196 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.104046 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-tls\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.104196 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.104129 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-2\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.104196 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.104158 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-thanos-prometheus-http-client-file\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.104196 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.104159 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:00:15.104196 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.104188 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.104598 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.104214 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-metrics-client-ca\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.104598 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.104268 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-0\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.104598 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.104314 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-db\") pod \"52227af2-b42c-4796-ac5a-43dca42cd06d\" (UID: \"52227af2-b42c-4796-ac5a-43dca42cd06d\") " Sep 06 14:00:15.104796 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.104772 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.104867 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.104806 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.106201 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.105268 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:00:15.106201 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.105389 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:00:15.106201 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.105686 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:00:15.107352 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.107315 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:15.107590 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.107474 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52227af2-b42c-4796-ac5a-43dca42cd06d-kube-api-access-4krxg" (OuterVolumeSpecName: "kube-api-access-4krxg") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "kube-api-access-4krxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:00:15.107734 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.107668 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:00:15.108070 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.107954 2577 scope.go:117] "RemoveContainer" containerID="6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857" Sep 06 14:00:15.108070 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.108023 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:00:15.108475 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.108337 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:00:15.108475 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.108349 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:15.108736 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.108703 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52227af2-b42c-4796-ac5a-43dca42cd06d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:00:15.108809 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.108769 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:15.109131 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.109110 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52227af2-b42c-4796-ac5a-43dca42cd06d-config-out" (OuterVolumeSpecName: "config-out") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:00:15.109621 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.109585 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-config" (OuterVolumeSpecName: "config") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:15.109783 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.109747 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:15.109856 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.109816 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:15.109935 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.109885 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:15.110474 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.110454 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:15.115580 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.115561 2577 scope.go:117] "RemoveContainer" containerID="686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e" Sep 06 14:00:15.119998 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.119900 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-web-config" (OuterVolumeSpecName: "web-config") pod "52227af2-b42c-4796-ac5a-43dca42cd06d" (UID: "52227af2-b42c-4796-ac5a-43dca42cd06d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:15.122666 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.122650 2577 scope.go:117] "RemoveContainer" containerID="e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17" Sep 06 14:00:15.129422 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.129405 2577 scope.go:117] "RemoveContainer" containerID="a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3" Sep 06 14:00:15.129672 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:00:15.129655 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3\": container with ID starting with a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3 not found: ID does not exist" containerID="a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3" Sep 06 14:00:15.129715 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.129683 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3"} err="failed to get container status \"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3\": rpc error: code = NotFound desc = could not find container \"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3\": container with ID starting with a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3 not found: ID does not exist" Sep 06 14:00:15.129715 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.129703 2577 scope.go:117] "RemoveContainer" containerID="627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2" Sep 06 14:00:15.129981 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:00:15.129961 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2\": container with ID starting with 627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2 not found: ID does not exist" containerID="627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2" Sep 06 14:00:15.130029 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.129986 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2"} err="failed to get container status \"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2\": rpc error: code = NotFound desc = could not find container \"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2\": container with ID starting with 627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2 not found: ID does not exist" Sep 06 14:00:15.130029 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.130002 2577 scope.go:117] "RemoveContainer" containerID="4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925" Sep 06 14:00:15.130239 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:00:15.130224 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925\": container with ID starting with 4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925 not found: ID does not exist" containerID="4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925" Sep 06 14:00:15.130275 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.130243 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925"} err="failed to get container status \"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925\": rpc error: code = NotFound desc = could not find container \"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925\": container with ID starting with 4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925 not found: ID does not exist" Sep 06 14:00:15.130275 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.130257 2577 scope.go:117] "RemoveContainer" containerID="81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1" Sep 06 14:00:15.130493 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:00:15.130477 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1\": container with ID starting with 81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1 not found: ID does not exist" containerID="81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1" Sep 06 14:00:15.130539 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.130495 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1"} err="failed to get container status \"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1\": rpc error: code = NotFound desc = could not find container \"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1\": container with ID starting with 81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1 not found: ID does not exist" Sep 06 14:00:15.130539 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.130511 2577 scope.go:117] "RemoveContainer" containerID="6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857" Sep 06 14:00:15.130757 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:00:15.130741 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857\": container with ID starting with 6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857 not found: ID does not exist" containerID="6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857" Sep 06 14:00:15.130793 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.130763 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857"} err="failed to get container status \"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857\": rpc error: code = NotFound desc = could not find container \"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857\": container with ID starting with 6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857 not found: ID does not exist" Sep 06 14:00:15.130793 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.130777 2577 scope.go:117] "RemoveContainer" containerID="686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e" Sep 06 14:00:15.131034 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:00:15.131019 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e\": container with ID starting with 686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e not found: ID does not exist" containerID="686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e" Sep 06 14:00:15.131091 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.131037 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e"} err="failed to get container status \"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e\": rpc error: code = NotFound desc = could not find container \"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e\": container with ID starting with 686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e not found: ID does not exist" Sep 06 14:00:15.131091 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.131050 2577 scope.go:117] "RemoveContainer" containerID="e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17" Sep 06 14:00:15.131270 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:00:15.131255 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17\": container with ID starting with e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17 not found: ID does not exist" containerID="e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17" Sep 06 14:00:15.131305 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.131273 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17"} err="failed to get container status \"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17\": rpc error: code = NotFound desc = could not find container \"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17\": container with ID starting with e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17 not found: ID does not exist" Sep 06 14:00:15.131305 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.131287 2577 scope.go:117] "RemoveContainer" containerID="a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3" Sep 06 14:00:15.131496 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.131482 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3"} err="failed to get container status \"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3\": rpc error: code = NotFound desc = could not find container \"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3\": container with ID starting with a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3 not found: ID does not exist" Sep 06 14:00:15.131496 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.131495 2577 scope.go:117] "RemoveContainer" containerID="627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2" Sep 06 14:00:15.131693 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.131669 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2"} err="failed to get container status \"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2\": rpc error: code = NotFound desc = could not find container \"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2\": container with ID starting with 627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2 not found: ID does not exist" Sep 06 14:00:15.131730 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.131693 2577 scope.go:117] "RemoveContainer" containerID="4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925" Sep 06 14:00:15.131876 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.131862 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925"} err="failed to get container status \"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925\": rpc error: code = NotFound desc = could not find container \"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925\": container with ID starting with 4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925 not found: ID does not exist" Sep 06 14:00:15.131876 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.131875 2577 scope.go:117] "RemoveContainer" containerID="81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1" Sep 06 14:00:15.132079 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.132061 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1"} err="failed to get container status \"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1\": rpc error: code = NotFound desc = could not find container \"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1\": container with ID starting with 81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1 not found: ID does not exist" Sep 06 14:00:15.132125 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.132079 2577 scope.go:117] "RemoveContainer" containerID="6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857" Sep 06 14:00:15.132328 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.132312 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857"} err="failed to get container status \"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857\": rpc error: code = NotFound desc = could not find container \"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857\": container with ID starting with 6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857 not found: ID does not exist" Sep 06 14:00:15.132328 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.132327 2577 scope.go:117] "RemoveContainer" containerID="686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e" Sep 06 14:00:15.132534 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.132519 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e"} err="failed to get container status \"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e\": rpc error: code = NotFound desc = could not find container \"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e\": container with ID starting with 686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e not found: ID does not exist" Sep 06 14:00:15.132581 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.132534 2577 scope.go:117] "RemoveContainer" containerID="e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17" Sep 06 14:00:15.132756 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.132739 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17"} err="failed to get container status \"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17\": rpc error: code = NotFound desc = could not find container \"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17\": container with ID starting with e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17 not found: ID does not exist" Sep 06 14:00:15.132798 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.132757 2577 scope.go:117] "RemoveContainer" containerID="a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3" Sep 06 14:00:15.132991 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.132976 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3"} err="failed to get container status \"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3\": rpc error: code = NotFound desc = could not find container \"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3\": container with ID starting with a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3 not found: ID does not exist" Sep 06 14:00:15.133030 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.132991 2577 scope.go:117] "RemoveContainer" containerID="627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2" Sep 06 14:00:15.133221 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.133203 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2"} err="failed to get container status \"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2\": rpc error: code = NotFound desc = could not find container \"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2\": container with ID starting with 627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2 not found: ID does not exist" Sep 06 14:00:15.133277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.133222 2577 scope.go:117] "RemoveContainer" containerID="4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925" Sep 06 14:00:15.133415 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.133399 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925"} err="failed to get container status \"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925\": rpc error: code = NotFound desc = could not find container \"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925\": container with ID starting with 4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925 not found: ID does not exist" Sep 06 14:00:15.133458 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.133415 2577 scope.go:117] "RemoveContainer" containerID="81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1" Sep 06 14:00:15.133628 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.133610 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1"} err="failed to get container status \"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1\": rpc error: code = NotFound desc = could not find container \"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1\": container with ID starting with 81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1 not found: ID does not exist" Sep 06 14:00:15.133672 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.133627 2577 scope.go:117] "RemoveContainer" containerID="6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857" Sep 06 14:00:15.133832 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.133813 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857"} err="failed to get container status \"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857\": rpc error: code = NotFound desc = could not find container \"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857\": container with ID starting with 6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857 not found: ID does not exist" Sep 06 14:00:15.133876 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.133833 2577 scope.go:117] "RemoveContainer" containerID="686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e" Sep 06 14:00:15.134064 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.134050 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e"} err="failed to get container status \"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e\": rpc error: code = NotFound desc = could not find container \"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e\": container with ID starting with 686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e not found: ID does not exist" Sep 06 14:00:15.134064 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.134063 2577 scope.go:117] "RemoveContainer" containerID="e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17" Sep 06 14:00:15.134276 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.134260 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17"} err="failed to get container status \"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17\": rpc error: code = NotFound desc = could not find container \"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17\": container with ID starting with e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17 not found: ID does not exist" Sep 06 14:00:15.134318 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.134277 2577 scope.go:117] "RemoveContainer" containerID="a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3" Sep 06 14:00:15.134468 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.134452 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3"} err="failed to get container status \"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3\": rpc error: code = NotFound desc = could not find container \"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3\": container with ID starting with a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3 not found: ID does not exist" Sep 06 14:00:15.134512 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.134470 2577 scope.go:117] "RemoveContainer" containerID="627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2" Sep 06 14:00:15.134664 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.134647 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2"} err="failed to get container status \"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2\": rpc error: code = NotFound desc = could not find container \"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2\": container with ID starting with 627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2 not found: ID does not exist" Sep 06 14:00:15.134702 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.134665 2577 scope.go:117] "RemoveContainer" containerID="4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925" Sep 06 14:00:15.134894 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.134876 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925"} err="failed to get container status \"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925\": rpc error: code = NotFound desc = could not find container \"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925\": container with ID starting with 4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925 not found: ID does not exist" Sep 06 14:00:15.134973 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.134895 2577 scope.go:117] "RemoveContainer" containerID="81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1" Sep 06 14:00:15.135166 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.135150 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1"} err="failed to get container status \"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1\": rpc error: code = NotFound desc = could not find container \"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1\": container with ID starting with 81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1 not found: ID does not exist" Sep 06 14:00:15.135207 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.135167 2577 scope.go:117] "RemoveContainer" containerID="6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857" Sep 06 14:00:15.135351 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.135335 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857"} err="failed to get container status \"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857\": rpc error: code = NotFound desc = could not find container \"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857\": container with ID starting with 6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857 not found: ID does not exist" Sep 06 14:00:15.135391 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.135351 2577 scope.go:117] "RemoveContainer" containerID="686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e" Sep 06 14:00:15.135577 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.135559 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e"} err="failed to get container status \"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e\": rpc error: code = NotFound desc = could not find container \"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e\": container with ID starting with 686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e not found: ID does not exist" Sep 06 14:00:15.135623 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.135578 2577 scope.go:117] "RemoveContainer" containerID="e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17" Sep 06 14:00:15.135794 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.135778 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17"} err="failed to get container status \"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17\": rpc error: code = NotFound desc = could not find container \"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17\": container with ID starting with e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17 not found: ID does not exist" Sep 06 14:00:15.135831 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.135796 2577 scope.go:117] "RemoveContainer" containerID="a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3" Sep 06 14:00:15.136027 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.136012 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3"} err="failed to get container status \"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3\": rpc error: code = NotFound desc = could not find container \"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3\": container with ID starting with a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3 not found: ID does not exist" Sep 06 14:00:15.136075 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.136026 2577 scope.go:117] "RemoveContainer" containerID="627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2" Sep 06 14:00:15.136241 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.136223 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2"} err="failed to get container status \"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2\": rpc error: code = NotFound desc = could not find container \"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2\": container with ID starting with 627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2 not found: ID does not exist" Sep 06 14:00:15.136283 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.136242 2577 scope.go:117] "RemoveContainer" containerID="4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925" Sep 06 14:00:15.136430 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.136417 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925"} err="failed to get container status \"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925\": rpc error: code = NotFound desc = could not find container \"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925\": container with ID starting with 4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925 not found: ID does not exist" Sep 06 14:00:15.136466 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.136431 2577 scope.go:117] "RemoveContainer" containerID="81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1" Sep 06 14:00:15.136642 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.136627 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1"} err="failed to get container status \"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1\": rpc error: code = NotFound desc = could not find container \"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1\": container with ID starting with 81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1 not found: ID does not exist" Sep 06 14:00:15.136683 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.136643 2577 scope.go:117] "RemoveContainer" containerID="6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857" Sep 06 14:00:15.136864 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.136845 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857"} err="failed to get container status \"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857\": rpc error: code = NotFound desc = could not find container \"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857\": container with ID starting with 6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857 not found: ID does not exist" Sep 06 14:00:15.136864 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.136863 2577 scope.go:117] "RemoveContainer" containerID="686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e" Sep 06 14:00:15.137079 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.137061 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e"} err="failed to get container status \"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e\": rpc error: code = NotFound desc = could not find container \"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e\": container with ID starting with 686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e not found: ID does not exist" Sep 06 14:00:15.137116 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.137079 2577 scope.go:117] "RemoveContainer" containerID="e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17" Sep 06 14:00:15.137291 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.137274 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17"} err="failed to get container status \"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17\": rpc error: code = NotFound desc = could not find container \"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17\": container with ID starting with e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17 not found: ID does not exist" Sep 06 14:00:15.137345 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.137292 2577 scope.go:117] "RemoveContainer" containerID="a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3" Sep 06 14:00:15.137488 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.137470 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3"} err="failed to get container status \"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3\": rpc error: code = NotFound desc = could not find container \"a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3\": container with ID starting with a6dc459f44c457d6afd56a0fd25e4efb8f2ec165e73ebce4743eea91f269a8c3 not found: ID does not exist" Sep 06 14:00:15.137535 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.137488 2577 scope.go:117] "RemoveContainer" containerID="627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2" Sep 06 14:00:15.137697 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.137678 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2"} err="failed to get container status \"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2\": rpc error: code = NotFound desc = could not find container \"627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2\": container with ID starting with 627be1401fa80c7b5ed81e3c1e4d3e8f80df6f30e4041ef2581244dd662e97a2 not found: ID does not exist" Sep 06 14:00:15.137738 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.137697 2577 scope.go:117] "RemoveContainer" containerID="4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925" Sep 06 14:00:15.137919 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.137889 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925"} err="failed to get container status \"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925\": rpc error: code = NotFound desc = could not find container \"4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925\": container with ID starting with 4ed0858ae0d917803f41d2026e249fa250a0f19a1de039a3efbf3d376681c925 not found: ID does not exist" Sep 06 14:00:15.137964 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.137930 2577 scope.go:117] "RemoveContainer" containerID="81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1" Sep 06 14:00:15.138169 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.138154 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1"} err="failed to get container status \"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1\": rpc error: code = NotFound desc = could not find container \"81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1\": container with ID starting with 81c39fde0ee82cde6203739b25705cdf52634c27c849f42d001fa83859079ef1 not found: ID does not exist" Sep 06 14:00:15.138213 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.138169 2577 scope.go:117] "RemoveContainer" containerID="6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857" Sep 06 14:00:15.138383 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.138366 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857"} err="failed to get container status \"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857\": rpc error: code = NotFound desc = could not find container \"6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857\": container with ID starting with 6e10baa9d60982d73768e4b67865297ece2a6684f0c01430aa97d1c3b5f53857 not found: ID does not exist" Sep 06 14:00:15.138427 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.138383 2577 scope.go:117] "RemoveContainer" containerID="686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e" Sep 06 14:00:15.138590 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.138572 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e"} err="failed to get container status \"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e\": rpc error: code = NotFound desc = could not find container \"686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e\": container with ID starting with 686db14ea8f38c90dd86ca7c3544df4546d2701ebaf189c36eb3898e4c5c373e not found: ID does not exist" Sep 06 14:00:15.138638 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.138590 2577 scope.go:117] "RemoveContainer" containerID="e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17" Sep 06 14:00:15.138790 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.138772 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17"} err="failed to get container status \"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17\": rpc error: code = NotFound desc = could not find container \"e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17\": container with ID starting with e149ad75fccc8815b531bbea446ee9e3132077690720687e63641ecc82a98c17 not found: ID does not exist" Sep 06 14:00:15.205395 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205369 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205395 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205396 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4krxg\" (UniqueName: \"kubernetes.io/projected/52227af2-b42c-4796-ac5a-43dca42cd06d-kube-api-access-4krxg\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205408 2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-grpc-tls\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205417 2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-kube-rbac-proxy\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205428 2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52227af2-b42c-4796-ac5a-43dca42cd06d-config-out\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205436 2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52227af2-b42c-4796-ac5a-43dca42cd06d-tls-assets\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205443 2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-web-config\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205450 2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-config\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205458 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205466 2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-metrics-client-certs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205475 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205484 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205492 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205500 2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205508 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52227af2-b42c-4796-ac5a-43dca42cd06d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205517 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-configmap-metrics-client-ca\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205526 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.205556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.205533 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/52227af2-b42c-4796-ac5a-43dca42cd06d-prometheus-k8s-db\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:00:15.395140 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.395104 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 14:00:15.399122 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.399097 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 14:00:15.422590 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.422566 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 14:00:15.422877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.422865 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="thanos-sidecar" Sep 06 14:00:15.422927 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.422879 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="thanos-sidecar" Sep 06 14:00:15.422927 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.422892 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="init-config-reloader" Sep 06 14:00:15.422927 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.422898 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="init-config-reloader" Sep 06 14:00:15.422927 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.422925 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="prometheus" Sep 06 14:00:15.423057 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.422932 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="prometheus" Sep 06 14:00:15.423057 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.422942 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="config-reloader" Sep 06 14:00:15.423057 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.422948 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="config-reloader" Sep 06 14:00:15.423057 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.422954 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="kube-rbac-proxy-web" Sep 06 14:00:15.423057 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.422960 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="kube-rbac-proxy-web" Sep 06 14:00:15.423057 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.422967 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="kube-rbac-proxy-thanos" Sep 06 14:00:15.423057 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.422972 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="kube-rbac-proxy-thanos" Sep 06 14:00:15.423057 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.422983 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="kube-rbac-proxy" Sep 06 14:00:15.423057 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.422988 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="kube-rbac-proxy" Sep 06 14:00:15.423057 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.423033 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="kube-rbac-proxy" Sep 06 14:00:15.423057 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.423043 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="prometheus" Sep 06 14:00:15.423057 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.423049 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="thanos-sidecar" Sep 06 14:00:15.423057 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.423055 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="kube-rbac-proxy-thanos" Sep 06 14:00:15.423057 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.423060 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="kube-rbac-proxy-web" Sep 06 14:00:15.423424 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.423065 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" containerName="config-reloader" Sep 06 14:00:15.427320 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.427277 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.430371 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.430349 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 06 14:00:15.430465 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.430349 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7q8vk\"" Sep 06 14:00:15.430465 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.430349 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 06 14:00:15.430667 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.430650 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 06 14:00:15.430739 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.430674 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 06 14:00:15.430739 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.430683 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 06 14:00:15.430828 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.430737 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 06 14:00:15.431213 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.431077 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 06 14:00:15.431213 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.431170 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 06 14:00:15.431213 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.431178 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 06 14:00:15.431434 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.431235 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 06 14:00:15.431434 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.431269 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 06 14:00:15.431583 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.431565 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 06 14:00:15.431643 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.431568 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9fq4mll5g11dm\"" Sep 06 14:00:15.433599 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.433581 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 06 14:00:15.436220 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.436202 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 06 14:00:15.439948 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.439926 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 14:00:15.507790 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.507763 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.507790 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.507792 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508017 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.507818 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508017 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.507856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508017 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.507887 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c1307f0-e605-4d79-9f62-dda3c55b0881-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508017 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.507931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c1307f0-e605-4d79-9f62-dda3c55b0881-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508017 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.507946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508017 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.507963 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508017 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.507981 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c1307f0-e605-4d79-9f62-dda3c55b0881-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508017 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.508013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508258 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.508034 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-config\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508258 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.508056 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmvk2\" (UniqueName: \"kubernetes.io/projected/7c1307f0-e605-4d79-9f62-dda3c55b0881-kube-api-access-lmvk2\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508258 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.508107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508258 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.508133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508258 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.508167 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508258 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.508183 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508258 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.508211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508258 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.508226 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508258 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.508246 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.508522 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.508264 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.609586 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.609557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.609724 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.609598 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.609724 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.609633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.609724 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.609662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.609724 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.609690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.609724 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.609714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.609993 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.609753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.609993 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.609777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.609993 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.609803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c1307f0-e605-4d79-9f62-dda3c55b0881-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.609993 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.609836 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c1307f0-e605-4d79-9f62-dda3c55b0881-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.609993 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.609859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.609993 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.609883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.609993 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.609927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c1307f0-e605-4d79-9f62-dda3c55b0881-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.609993 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.609956 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.609993 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.609990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-config\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.610435 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.610026 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmvk2\" (UniqueName: \"kubernetes.io/projected/7c1307f0-e605-4d79-9f62-dda3c55b0881-kube-api-access-lmvk2\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.610435 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.610065 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.610435 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.610096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.610435 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.610142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.610435 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.610167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.610435 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.610370 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.611047 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.610926 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.611122 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.611066 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c1307f0-e605-4d79-9f62-dda3c55b0881-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.612227 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.611236 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.612227 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.611431 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.612227 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.611878 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.612993 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.612968 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.613343 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.613319 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.614471 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.614442 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c1307f0-e605-4d79-9f62-dda3c55b0881-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.614756 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.614582 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-config\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.614875 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.614850 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c1307f0-e605-4d79-9f62-dda3c55b0881-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.615114 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.615089 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c1307f0-e605-4d79-9f62-dda3c55b0881-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.615521 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.615500 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.615808 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.615757 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.616195 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.616174 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.616297 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.616201 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.616855 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.616836 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.617338 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.617315 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.617662 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.617641 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c1307f0-e605-4d79-9f62-dda3c55b0881-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.619369 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.619353 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmvk2\" (UniqueName: \"kubernetes.io/projected/7c1307f0-e605-4d79-9f62-dda3c55b0881-kube-api-access-lmvk2\") pod \"prometheus-k8s-0\" (UID: \"7c1307f0-e605-4d79-9f62-dda3c55b0881\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:15.738891 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:15.738799 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:16.023777 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:16.023739 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 14:00:16.028237 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:00:16.028202 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1307f0_e605_4d79_9f62_dda3c55b0881.slice/crio-4161f1ebd87c210efbdd7fdf248cecc7bf2b1a7c38ca1931f88ded785719cb23 WatchSource:0}: Error finding container 4161f1ebd87c210efbdd7fdf248cecc7bf2b1a7c38ca1931f88ded785719cb23: Status 404 returned error can't find the container with id 4161f1ebd87c210efbdd7fdf248cecc7bf2b1a7c38ca1931f88ded785719cb23 Sep 06 14:00:16.079075 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:16.079040 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" event={"ID":"c8b26446-fc0a-4422-b6fd-5771b7e9a14a","Type":"ContainerStarted","Data":"50ab2101890623957316fffb47027fd655b287b6613351c78961081f20aa3210"} Sep 06 14:00:16.080777 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:16.080749 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c1307f0-e605-4d79-9f62-dda3c55b0881","Type":"ContainerStarted","Data":"4161f1ebd87c210efbdd7fdf248cecc7bf2b1a7c38ca1931f88ded785719cb23"} Sep 06 14:00:17.067985 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:17.067956 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52227af2-b42c-4796-ac5a-43dca42cd06d" path="/var/lib/kubelet/pods/52227af2-b42c-4796-ac5a-43dca42cd06d/volumes" Sep 06 14:00:17.087202 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:17.087172 2577 generic.go:356] "Generic (PLEG): container finished" podID="7c1307f0-e605-4d79-9f62-dda3c55b0881" containerID="d79a5aedd11ef35910fe2ab59dc99d5e74e92e303bfb482a788c8fca105ffc52" exitCode=0 Sep 06 14:00:17.087548 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:17.087299 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c1307f0-e605-4d79-9f62-dda3c55b0881","Type":"ContainerDied","Data":"d79a5aedd11ef35910fe2ab59dc99d5e74e92e303bfb482a788c8fca105ffc52"} Sep 06 14:00:17.089923 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:17.089882 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" event={"ID":"c8b26446-fc0a-4422-b6fd-5771b7e9a14a","Type":"ContainerStarted","Data":"6eb700b808423a243bdfddb8da48c2c86fbcc38d7fc1191dc80e93659208a3b4"} Sep 06 14:00:17.090018 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:17.089962 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" event={"ID":"c8b26446-fc0a-4422-b6fd-5771b7e9a14a","Type":"ContainerStarted","Data":"a01cf8d3edc7c6e201aa8967b90d8bcda338bcdafa07a12acbd5f90e2c216901"} Sep 06 14:00:17.090018 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:17.089994 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" event={"ID":"c8b26446-fc0a-4422-b6fd-5771b7e9a14a","Type":"ContainerStarted","Data":"bd5e29a7e358a7802b9d2858f0b9668f1e33c1379da2cd945b7f6f7d2ecda0a2"} Sep 06 14:00:17.128768 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:17.128706 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-567955b447-sm9q6" podStartSLOduration=1.317420436 podStartE2EDuration="3.128685492s" totalImagesPullingTime="1.811265056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:00:17.127686466 +0000 UTC m=+148.583175348" watchObservedRunningTime="2026-09-06 14:00:17.128685492 +0000 UTC m=+148.584174362" Sep 06 14:00:17.253476 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:17.253440 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:00:18.097464 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:18.097426 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c1307f0-e605-4d79-9f62-dda3c55b0881","Type":"ContainerStarted","Data":"1907b05007593f975778a3caa57d6cccd993c57012f7f203df65ab9ce431a755"} Sep 06 14:00:18.097971 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:18.097546 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c1307f0-e605-4d79-9f62-dda3c55b0881","Type":"ContainerStarted","Data":"8960efd3d5c4217ea446d3dc209e532a9bd9c897024a149f2913a6bad86e0170"} Sep 06 14:00:18.097971 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:18.097597 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c1307f0-e605-4d79-9f62-dda3c55b0881","Type":"ContainerStarted","Data":"4b76b2cb0c68039accc052eb836401c6dc67738dc54e6e7da1156f2aedd55ed6"} Sep 06 14:00:18.097971 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:18.097645 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c1307f0-e605-4d79-9f62-dda3c55b0881","Type":"ContainerStarted","Data":"6728e4b8c136bb374411a5052d97104c506bed982768e03d6cd367e85a0f4b5d"} Sep 06 14:00:18.097971 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:18.097684 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c1307f0-e605-4d79-9f62-dda3c55b0881","Type":"ContainerStarted","Data":"03cfbad473ad52270658af75727ff7ffbd9a5d954b533cd900684979b3bf372f"} Sep 06 14:00:18.097971 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:18.097724 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c1307f0-e605-4d79-9f62-dda3c55b0881","Type":"ContainerStarted","Data":"32b804a3923682b83bcad3da26ed229e128949c52da2cca1b5ed7606d5cff653"} Sep 06 14:00:18.123113 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:18.123046 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.095680854 podStartE2EDuration="3.123025607s" totalImagesPullingTime="0s" totalInitContainerRuntime="27.344753ms" isStatefulPod=false podCreationTimestamp="2026-09-06 14:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:00:18.120723547 +0000 UTC m=+149.576212445" watchObservedRunningTime="2026-09-06 14:00:18.123025607 +0000 UTC m=+149.578514472" Sep 06 14:00:20.739665 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:20.739635 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:00:28.265804 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:28.265770 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:00:28.319517 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:28.319486 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:00:28.370845 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:00:28.370813 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:01:15.739120 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:01:15.739086 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:01:15.754737 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:01:15.754708 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:01:16.349081 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:01:16.349054 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 14:01:24.606181 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:01:24.606146 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:01:24.659436 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:01:24.659405 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:01:36.501863 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:01:36.501826 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:01:36.553846 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:01:36.553818 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:01:49.319341 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:01:49.319308 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:01:53.380014 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:01:53.379977 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:01:53.477903 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:01:53.477864 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:00.578197 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:00.578158 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:00.625019 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:00.624985 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:00.678076 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:00.678058 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:00.727623 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:00.727602 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:13.570984 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:13.570886 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:13.667068 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:13.667036 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:35.174151 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:35.174091 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:36.548486 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:36.548452 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:38.549072 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:38.549038 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:38.601059 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:38.601033 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:39.766507 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:39.766474 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:39.867324 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:39.867301 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:40.011368 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.011337 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:40.017818 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.017753 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5cfdv"] Sep 06 14:02:40.021202 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.021185 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5cfdv" Sep 06 14:02:40.023534 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.023515 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-pkfq7\"" Sep 06 14:02:40.023793 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.023778 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 06 14:02:40.024455 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.024438 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 06 14:02:40.033003 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.032980 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5cfdv"] Sep 06 14:02:40.113167 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.113132 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:40.180868 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.180841 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abc0793e-cc43-4b49-9e5c-da8b3e9e4fbc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5cfdv\" (UID: \"abc0793e-cc43-4b49-9e5c-da8b3e9e4fbc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5cfdv" Sep 06 14:02:40.181015 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.180876 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbs5r\" (UniqueName: \"kubernetes.io/projected/abc0793e-cc43-4b49-9e5c-da8b3e9e4fbc-kube-api-access-dbs5r\") pod \"cert-manager-webhook-6486ffd997-5cfdv\" (UID: \"abc0793e-cc43-4b49-9e5c-da8b3e9e4fbc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5cfdv" Sep 06 14:02:40.281656 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.281634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abc0793e-cc43-4b49-9e5c-da8b3e9e4fbc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5cfdv\" (UID: \"abc0793e-cc43-4b49-9e5c-da8b3e9e4fbc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5cfdv" Sep 06 14:02:40.281825 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.281664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbs5r\" (UniqueName: \"kubernetes.io/projected/abc0793e-cc43-4b49-9e5c-da8b3e9e4fbc-kube-api-access-dbs5r\") pod \"cert-manager-webhook-6486ffd997-5cfdv\" (UID: \"abc0793e-cc43-4b49-9e5c-da8b3e9e4fbc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5cfdv" Sep 06 14:02:40.295434 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.295411 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abc0793e-cc43-4b49-9e5c-da8b3e9e4fbc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5cfdv\" (UID: \"abc0793e-cc43-4b49-9e5c-da8b3e9e4fbc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5cfdv" Sep 06 14:02:40.295434 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.295425 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbs5r\" (UniqueName: \"kubernetes.io/projected/abc0793e-cc43-4b49-9e5c-da8b3e9e4fbc-kube-api-access-dbs5r\") pod \"cert-manager-webhook-6486ffd997-5cfdv\" (UID: \"abc0793e-cc43-4b49-9e5c-da8b3e9e4fbc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5cfdv" Sep 06 14:02:40.348061 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.348033 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5cfdv" Sep 06 14:02:40.621933 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.621874 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5cfdv"] Sep 06 14:02:40.624741 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:02:40.624709 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabc0793e_cc43_4b49_9e5c_da8b3e9e4fbc.slice/crio-8583002302bfd57ae8c18e9a5306fbdc8742521d5b7f8479dd6f4950c54a297c WatchSource:0}: Error finding container 8583002302bfd57ae8c18e9a5306fbdc8742521d5b7f8479dd6f4950c54a297c: Status 404 returned error can't find the container with id 8583002302bfd57ae8c18e9a5306fbdc8742521d5b7f8479dd6f4950c54a297c Sep 06 14:02:40.663886 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:40.663856 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5cfdv" event={"ID":"abc0793e-cc43-4b49-9e5c-da8b3e9e4fbc","Type":"ContainerStarted","Data":"8583002302bfd57ae8c18e9a5306fbdc8742521d5b7f8479dd6f4950c54a297c"} Sep 06 14:02:41.974653 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:41.974573 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:42.171543 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:42.171481 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:44.684385 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:44.684349 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5cfdv" event={"ID":"abc0793e-cc43-4b49-9e5c-da8b3e9e4fbc","Type":"ContainerStarted","Data":"89767ac8c9defc03bd98023865bdff35004350494f5eab6fc2dc50920c8fd9a3"} Sep 06 14:02:44.684824 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:44.684476 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5cfdv" Sep 06 14:02:44.699211 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:44.699164 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5cfdv" podStartSLOduration=2.538207908 podStartE2EDuration="5.699147927s" totalImagesPullingTime="3.160940019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:02:44.697622271 +0000 UTC m=+296.153111134" watchObservedRunningTime="2026-09-06 14:02:44.699147927 +0000 UTC m=+296.154636788" Sep 06 14:02:45.151949 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:45.151893 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:45.205493 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:45.205465 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:47.940581 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:47.940541 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:47.993373 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:47.993343 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:48.983838 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:48.983813 2577 kubelet.go:1648] "Image garbage collection succeeded" Sep 06 14:02:49.962325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:49.962297 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:50.691175 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:50.691144 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5cfdv" Sep 06 14:02:51.208707 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:51.208677 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tlmf4"] Sep 06 14:02:51.211871 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:51.211855 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tlmf4" Sep 06 14:02:51.214604 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:51.214583 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-kwrfs\"" Sep 06 14:02:51.219604 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:51.219581 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tlmf4"] Sep 06 14:02:51.373737 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:51.373702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7krd\" (UniqueName: \"kubernetes.io/projected/9751d86a-f3e2-4e4e-be6e-a8d84834a8bf-kube-api-access-v7krd\") pod \"cert-manager-55b4d954d5-tlmf4\" (UID: \"9751d86a-f3e2-4e4e-be6e-a8d84834a8bf\") " pod="cert-manager/cert-manager-55b4d954d5-tlmf4" Sep 06 14:02:51.373888 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:51.373809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9751d86a-f3e2-4e4e-be6e-a8d84834a8bf-bound-sa-token\") pod \"cert-manager-55b4d954d5-tlmf4\" (UID: \"9751d86a-f3e2-4e4e-be6e-a8d84834a8bf\") " pod="cert-manager/cert-manager-55b4d954d5-tlmf4" Sep 06 14:02:51.474346 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:51.474285 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9751d86a-f3e2-4e4e-be6e-a8d84834a8bf-bound-sa-token\") pod \"cert-manager-55b4d954d5-tlmf4\" (UID: \"9751d86a-f3e2-4e4e-be6e-a8d84834a8bf\") " pod="cert-manager/cert-manager-55b4d954d5-tlmf4" Sep 06 14:02:51.474455 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:51.474344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7krd\" (UniqueName: \"kubernetes.io/projected/9751d86a-f3e2-4e4e-be6e-a8d84834a8bf-kube-api-access-v7krd\") pod \"cert-manager-55b4d954d5-tlmf4\" (UID: \"9751d86a-f3e2-4e4e-be6e-a8d84834a8bf\") " pod="cert-manager/cert-manager-55b4d954d5-tlmf4" Sep 06 14:02:51.481858 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:51.481836 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9751d86a-f3e2-4e4e-be6e-a8d84834a8bf-bound-sa-token\") pod \"cert-manager-55b4d954d5-tlmf4\" (UID: \"9751d86a-f3e2-4e4e-be6e-a8d84834a8bf\") " pod="cert-manager/cert-manager-55b4d954d5-tlmf4" Sep 06 14:02:51.481974 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:51.481929 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7krd\" (UniqueName: \"kubernetes.io/projected/9751d86a-f3e2-4e4e-be6e-a8d84834a8bf-kube-api-access-v7krd\") pod \"cert-manager-55b4d954d5-tlmf4\" (UID: \"9751d86a-f3e2-4e4e-be6e-a8d84834a8bf\") " pod="cert-manager/cert-manager-55b4d954d5-tlmf4" Sep 06 14:02:51.521915 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:51.521892 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tlmf4" Sep 06 14:02:51.788104 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:51.788077 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tlmf4"] Sep 06 14:02:51.790806 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:02:51.790776 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9751d86a_f3e2_4e4e_be6e_a8d84834a8bf.slice/crio-052194cddf0a5f84fb8d80e5288226da5dbedec451dfe69b394bf81034563bdb WatchSource:0}: Error finding container 052194cddf0a5f84fb8d80e5288226da5dbedec451dfe69b394bf81034563bdb: Status 404 returned error can't find the container with id 052194cddf0a5f84fb8d80e5288226da5dbedec451dfe69b394bf81034563bdb Sep 06 14:02:52.334889 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:52.334851 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:52.433189 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:52.433155 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:52.718438 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:52.718364 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tlmf4" event={"ID":"9751d86a-f3e2-4e4e-be6e-a8d84834a8bf","Type":"ContainerStarted","Data":"976ca8833ab3213e102e5831f18a9ee7ec8ad01d76bff125fa0ce9a03c6e897a"} Sep 06 14:02:52.718438 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:52.718402 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tlmf4" event={"ID":"9751d86a-f3e2-4e4e-be6e-a8d84834a8bf","Type":"ContainerStarted","Data":"052194cddf0a5f84fb8d80e5288226da5dbedec451dfe69b394bf81034563bdb"} Sep 06 14:02:52.734674 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:52.734628 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-tlmf4" podStartSLOduration=1.7345815039999999 podStartE2EDuration="1.734581504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:02:52.734038918 +0000 UTC m=+304.189527783" watchObservedRunningTime="2026-09-06 14:02:52.734581504 +0000 UTC m=+304.190070369" Sep 06 14:02:57.374215 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:57.374180 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:57.474325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:57.474297 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:02:59.323176 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:02:59.323145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:15.782635 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:15.782589 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:15.877327 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:15.877299 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:16.384972 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.384942 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:16.578213 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.578182 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:16.700078 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.699990 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:16.768430 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.768398 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s"] Sep 06 14:03:16.784406 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.784370 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.785885 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.785863 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s"] Sep 06 14:03:16.786694 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.786635 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 06 14:03:16.786844 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.786776 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 06 14:03:16.786939 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.786838 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 06 14:03:16.787066 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.786930 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-c7zg9\"" Sep 06 14:03:16.787131 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.787112 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 06 14:03:16.863102 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.863076 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4f783ae-85c9-4ab0-aee3-ca3573820243-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-zgl8s\" (UID: \"a4f783ae-85c9-4ab0-aee3-ca3573820243\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.863222 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.863110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a4f783ae-85c9-4ab0-aee3-ca3573820243-manifests\") pod \"opendatahub-operator-controller-manager-7b6858f974-zgl8s\" (UID: \"a4f783ae-85c9-4ab0-aee3-ca3573820243\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.863222 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.863128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rh4\" (UniqueName: \"kubernetes.io/projected/a4f783ae-85c9-4ab0-aee3-ca3573820243-kube-api-access-26rh4\") pod \"opendatahub-operator-controller-manager-7b6858f974-zgl8s\" (UID: \"a4f783ae-85c9-4ab0-aee3-ca3573820243\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.863222 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.863168 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4f783ae-85c9-4ab0-aee3-ca3573820243-tmp\") pod \"opendatahub-operator-controller-manager-7b6858f974-zgl8s\" (UID: \"a4f783ae-85c9-4ab0-aee3-ca3573820243\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.863348 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.863264 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a4f783ae-85c9-4ab0-aee3-ca3573820243-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-zgl8s\" (UID: \"a4f783ae-85c9-4ab0-aee3-ca3573820243\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.881678 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.881654 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:16.964027 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.963954 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a4f783ae-85c9-4ab0-aee3-ca3573820243-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-zgl8s\" (UID: \"a4f783ae-85c9-4ab0-aee3-ca3573820243\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.964027 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.964005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4f783ae-85c9-4ab0-aee3-ca3573820243-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-zgl8s\" (UID: \"a4f783ae-85c9-4ab0-aee3-ca3573820243\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.964230 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.964130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a4f783ae-85c9-4ab0-aee3-ca3573820243-manifests\") pod \"opendatahub-operator-controller-manager-7b6858f974-zgl8s\" (UID: \"a4f783ae-85c9-4ab0-aee3-ca3573820243\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.964230 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.964169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rh4\" (UniqueName: \"kubernetes.io/projected/a4f783ae-85c9-4ab0-aee3-ca3573820243-kube-api-access-26rh4\") pod \"opendatahub-operator-controller-manager-7b6858f974-zgl8s\" (UID: \"a4f783ae-85c9-4ab0-aee3-ca3573820243\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.964230 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.964193 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4f783ae-85c9-4ab0-aee3-ca3573820243-tmp\") pod \"opendatahub-operator-controller-manager-7b6858f974-zgl8s\" (UID: \"a4f783ae-85c9-4ab0-aee3-ca3573820243\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.964538 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.964503 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a4f783ae-85c9-4ab0-aee3-ca3573820243-manifests\") pod \"opendatahub-operator-controller-manager-7b6858f974-zgl8s\" (UID: \"a4f783ae-85c9-4ab0-aee3-ca3573820243\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.966477 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.966456 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a4f783ae-85c9-4ab0-aee3-ca3573820243-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-zgl8s\" (UID: \"a4f783ae-85c9-4ab0-aee3-ca3573820243\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.966477 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.966468 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4f783ae-85c9-4ab0-aee3-ca3573820243-tmp\") pod \"opendatahub-operator-controller-manager-7b6858f974-zgl8s\" (UID: \"a4f783ae-85c9-4ab0-aee3-ca3573820243\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.966705 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.966689 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4f783ae-85c9-4ab0-aee3-ca3573820243-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-zgl8s\" (UID: \"a4f783ae-85c9-4ab0-aee3-ca3573820243\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.972794 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.972777 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rh4\" (UniqueName: \"kubernetes.io/projected/a4f783ae-85c9-4ab0-aee3-ca3573820243-kube-api-access-26rh4\") pod \"opendatahub-operator-controller-manager-7b6858f974-zgl8s\" (UID: \"a4f783ae-85c9-4ab0-aee3-ca3573820243\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:16.978631 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:16.978588 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:17.078466 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:17.078420 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:17.098054 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:17.098028 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:17.186046 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:17.186013 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:17.288867 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:17.288828 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:17.386226 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:17.386195 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s"] Sep 06 14:03:17.390068 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:03:17.390039 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f783ae_85c9_4ab0_aee3_ca3573820243.slice/crio-6c53256c3c1e09630e4c0656715edc7108d6753975f4dc65f0bf2a534212cb1f WatchSource:0}: Error finding container 6c53256c3c1e09630e4c0656715edc7108d6753975f4dc65f0bf2a534212cb1f: Status 404 returned error can't find the container with id 6c53256c3c1e09630e4c0656715edc7108d6753975f4dc65f0bf2a534212cb1f Sep 06 14:03:17.392597 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:17.392572 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 14:03:17.821515 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:17.821482 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" event={"ID":"a4f783ae-85c9-4ab0-aee3-ca3573820243","Type":"ContainerStarted","Data":"6c53256c3c1e09630e4c0656715edc7108d6753975f4dc65f0bf2a534212cb1f"} Sep 06 14:03:19.608258 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:19.608226 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:19.704356 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:19.704322 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:20.839696 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:20.839656 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" event={"ID":"a4f783ae-85c9-4ab0-aee3-ca3573820243","Type":"ContainerStarted","Data":"b492c27836bc8aa3f0f15591d9273ea04b87aa6c3cfa8c68a34a144842a7786e"} Sep 06 14:03:21.845789 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:21.845756 2577 generic.go:356] "Generic (PLEG): container finished" podID="a4f783ae-85c9-4ab0-aee3-ca3573820243" containerID="b492c27836bc8aa3f0f15591d9273ea04b87aa6c3cfa8c68a34a144842a7786e" exitCode=0 Sep 06 14:03:21.846202 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:21.845813 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" event={"ID":"a4f783ae-85c9-4ab0-aee3-ca3573820243","Type":"ContainerDied","Data":"b492c27836bc8aa3f0f15591d9273ea04b87aa6c3cfa8c68a34a144842a7786e"} Sep 06 14:03:22.854146 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:22.854109 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" event={"ID":"a4f783ae-85c9-4ab0-aee3-ca3573820243","Type":"ContainerStarted","Data":"cb970e3a73f89af7676b2ec2be60f77e78a724ab6a00340eec2528660024b9f1"} Sep 06 14:03:22.854567 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:22.854252 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:22.875510 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:22.875465 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" podStartSLOduration=2.642218033 podStartE2EDuration="6.875448467s" totalImagesPullingTime="3.207082859s" totalInitContainerRuntime="1.026147575s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:03:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:03:22.873023669 +0000 UTC m=+334.328512533" watchObservedRunningTime="2026-09-06 14:03:22.875448467 +0000 UTC m=+334.330937326" Sep 06 14:03:28.371459 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:28.371411 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:28.470844 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:28.470802 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:33.861064 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:33.861035 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-zgl8s" Sep 06 14:03:35.418929 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:35.418872 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:35.514529 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:35.514485 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:35.743619 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:35.743501 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:36.213287 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:36.213252 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:36.953408 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:36.953375 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:37.023670 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:37.023618 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:37.755870 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:37.755832 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:42.769127 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:42.769058 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:42.869554 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:42.869515 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:43.052688 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:43.052580 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:43.164602 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:43.164574 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:43.651928 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:43.651869 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:46.469297 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.469257 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cwsj8"] Sep 06 14:03:46.472718 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.472698 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-cwsj8" Sep 06 14:03:46.474959 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.474931 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-hwr7c\"" Sep 06 14:03:46.475092 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.474982 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 06 14:03:46.480758 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.480733 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cwsj8"] Sep 06 14:03:46.499496 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.499465 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:46.606003 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.605968 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:46.609171 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.609152 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4d5ba38-49cd-4124-a20d-fbb5a0d27506-cert\") pod \"odh-model-controller-7998c67dc4-cwsj8\" (UID: \"a4d5ba38-49cd-4124-a20d-fbb5a0d27506\") " pod="opendatahub/odh-model-controller-7998c67dc4-cwsj8" Sep 06 14:03:46.609245 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.609198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4nbq\" (UniqueName: \"kubernetes.io/projected/a4d5ba38-49cd-4124-a20d-fbb5a0d27506-kube-api-access-n4nbq\") pod \"odh-model-controller-7998c67dc4-cwsj8\" (UID: \"a4d5ba38-49cd-4124-a20d-fbb5a0d27506\") " pod="opendatahub/odh-model-controller-7998c67dc4-cwsj8" Sep 06 14:03:46.698537 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.698504 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:46.710488 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.710459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4d5ba38-49cd-4124-a20d-fbb5a0d27506-cert\") pod \"odh-model-controller-7998c67dc4-cwsj8\" (UID: \"a4d5ba38-49cd-4124-a20d-fbb5a0d27506\") " pod="opendatahub/odh-model-controller-7998c67dc4-cwsj8" Sep 06 14:03:46.710624 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.710516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4nbq\" (UniqueName: \"kubernetes.io/projected/a4d5ba38-49cd-4124-a20d-fbb5a0d27506-kube-api-access-n4nbq\") pod \"odh-model-controller-7998c67dc4-cwsj8\" (UID: \"a4d5ba38-49cd-4124-a20d-fbb5a0d27506\") " pod="opendatahub/odh-model-controller-7998c67dc4-cwsj8" Sep 06 14:03:46.712873 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.712856 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4d5ba38-49cd-4124-a20d-fbb5a0d27506-cert\") pod \"odh-model-controller-7998c67dc4-cwsj8\" (UID: \"a4d5ba38-49cd-4124-a20d-fbb5a0d27506\") " pod="opendatahub/odh-model-controller-7998c67dc4-cwsj8" Sep 06 14:03:46.719419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.719367 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4nbq\" (UniqueName: \"kubernetes.io/projected/a4d5ba38-49cd-4124-a20d-fbb5a0d27506-kube-api-access-n4nbq\") pod \"odh-model-controller-7998c67dc4-cwsj8\" (UID: \"a4d5ba38-49cd-4124-a20d-fbb5a0d27506\") " pod="opendatahub/odh-model-controller-7998c67dc4-cwsj8" Sep 06 14:03:46.784380 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.784347 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-cwsj8" Sep 06 14:03:46.799257 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.799231 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:46.932138 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:46.932102 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:47.055721 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:47.055687 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cwsj8"] Sep 06 14:03:47.060705 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:03:47.060673 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d5ba38_49cd_4124_a20d_fbb5a0d27506.slice/crio-960d4e4942d1d1c186b523be91c94db41dfe8d3f74020be65ea5d0d71a5f291f WatchSource:0}: Error finding container 960d4e4942d1d1c186b523be91c94db41dfe8d3f74020be65ea5d0d71a5f291f: Status 404 returned error can't find the container with id 960d4e4942d1d1c186b523be91c94db41dfe8d3f74020be65ea5d0d71a5f291f Sep 06 14:03:47.325677 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:47.325589 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:47.971249 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:47.971211 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-cwsj8" event={"ID":"a4d5ba38-49cd-4124-a20d-fbb5a0d27506","Type":"ContainerStarted","Data":"960d4e4942d1d1c186b523be91c94db41dfe8d3f74020be65ea5d0d71a5f291f"} Sep 06 14:03:50.988687 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:50.988643 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-cwsj8" event={"ID":"a4d5ba38-49cd-4124-a20d-fbb5a0d27506","Type":"ContainerStarted","Data":"2d3568f2336270219e00d2f13fa93922fa07fdc5c4c82da1f1232dd0fd2ecda3"} Sep 06 14:03:50.989085 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:50.988784 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-cwsj8" Sep 06 14:03:51.004074 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:51.004027 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-cwsj8" podStartSLOduration=2.078906022 podStartE2EDuration="5.004012825s" totalImagesPullingTime="2.925106803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:03:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:03:51.001646853 +0000 UTC m=+362.457135715" watchObservedRunningTime="2026-09-06 14:03:51.004012825 +0000 UTC m=+362.459501688" Sep 06 14:03:53.627729 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.627694 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh"] Sep 06 14:03:53.631001 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.630976 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.633198 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.633178 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 06 14:03:53.634006 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.633979 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 06 14:03:53.634417 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.634403 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 06 14:03:53.635013 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.634996 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6nxqb\"" Sep 06 14:03:53.635322 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.634997 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 06 14:03:53.635661 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.635037 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 06 14:03:53.636032 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.634877 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 06 14:03:53.649499 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.649469 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh"] Sep 06 14:03:53.713729 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.713699 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-849f85686b-79g95"] Sep 06 14:03:53.716488 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.716467 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:03:53.719026 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.719004 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 06 14:03:53.719194 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.719048 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 06 14:03:53.719318 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.719095 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 06 14:03:53.719318 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.719107 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-q578l\"" Sep 06 14:03:53.719464 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.719122 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 06 14:03:53.719464 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.719136 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 06 14:03:53.727702 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.727684 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-849f85686b-79g95"] Sep 06 14:03:53.778159 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.778121 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/42eb9483-2a9e-4816-957c-5bfedefda1d2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.778324 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.778164 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/42eb9483-2a9e-4816-957c-5bfedefda1d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.778324 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.778235 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/42eb9483-2a9e-4816-957c-5bfedefda1d2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.778324 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.778265 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/42eb9483-2a9e-4816-957c-5bfedefda1d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.778456 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.778329 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42eb9483-2a9e-4816-957c-5bfedefda1d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.778456 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.778383 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42eb9483-2a9e-4816-957c-5bfedefda1d2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.778520 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.778464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbp52\" (UniqueName: \"kubernetes.io/projected/42eb9483-2a9e-4816-957c-5bfedefda1d2-kube-api-access-wbp52\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.879136 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.879059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42eb9483-2a9e-4816-957c-5bfedefda1d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.879136 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.879103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed50ed7c-3871-439c-9d3e-e07b609fefaa-cert\") pod \"lws-controller-manager-849f85686b-79g95\" (UID: \"ed50ed7c-3871-439c-9d3e-e07b609fefaa\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:03:53.879136 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.879126 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ed50ed7c-3871-439c-9d3e-e07b609fefaa-manager-config\") pod \"lws-controller-manager-849f85686b-79g95\" (UID: \"ed50ed7c-3871-439c-9d3e-e07b609fefaa\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:03:53.879341 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.879199 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42eb9483-2a9e-4816-957c-5bfedefda1d2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.879341 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.879235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbp52\" (UniqueName: \"kubernetes.io/projected/42eb9483-2a9e-4816-957c-5bfedefda1d2-kube-api-access-wbp52\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.879341 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.879252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqk6r\" (UniqueName: \"kubernetes.io/projected/ed50ed7c-3871-439c-9d3e-e07b609fefaa-kube-api-access-vqk6r\") pod \"lws-controller-manager-849f85686b-79g95\" (UID: \"ed50ed7c-3871-439c-9d3e-e07b609fefaa\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:03:53.879341 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.879306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/42eb9483-2a9e-4816-957c-5bfedefda1d2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.879341 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.879332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/42eb9483-2a9e-4816-957c-5bfedefda1d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.879563 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.879387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/42eb9483-2a9e-4816-957c-5bfedefda1d2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.879563 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.879418 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/42eb9483-2a9e-4816-957c-5bfedefda1d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.879563 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.879444 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed50ed7c-3871-439c-9d3e-e07b609fefaa-metrics-cert\") pod \"lws-controller-manager-849f85686b-79g95\" (UID: \"ed50ed7c-3871-439c-9d3e-e07b609fefaa\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:03:53.880277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.880237 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/42eb9483-2a9e-4816-957c-5bfedefda1d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.881527 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.881503 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42eb9483-2a9e-4816-957c-5bfedefda1d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.881637 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.881618 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/42eb9483-2a9e-4816-957c-5bfedefda1d2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.881747 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.881731 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/42eb9483-2a9e-4816-957c-5bfedefda1d2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.881785 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.881769 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/42eb9483-2a9e-4816-957c-5bfedefda1d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.886625 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.886600 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbp52\" (UniqueName: \"kubernetes.io/projected/42eb9483-2a9e-4816-957c-5bfedefda1d2-kube-api-access-wbp52\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.887175 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.887143 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42eb9483-2a9e-4816-957c-5bfedefda1d2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-wfwsh\" (UID: \"42eb9483-2a9e-4816-957c-5bfedefda1d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.945409 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.945384 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:53.979983 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.979946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqk6r\" (UniqueName: \"kubernetes.io/projected/ed50ed7c-3871-439c-9d3e-e07b609fefaa-kube-api-access-vqk6r\") pod \"lws-controller-manager-849f85686b-79g95\" (UID: \"ed50ed7c-3871-439c-9d3e-e07b609fefaa\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:03:53.980111 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.980055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed50ed7c-3871-439c-9d3e-e07b609fefaa-metrics-cert\") pod \"lws-controller-manager-849f85686b-79g95\" (UID: \"ed50ed7c-3871-439c-9d3e-e07b609fefaa\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:03:53.980111 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.980098 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed50ed7c-3871-439c-9d3e-e07b609fefaa-cert\") pod \"lws-controller-manager-849f85686b-79g95\" (UID: \"ed50ed7c-3871-439c-9d3e-e07b609fefaa\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:03:53.980188 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.980116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ed50ed7c-3871-439c-9d3e-e07b609fefaa-manager-config\") pod \"lws-controller-manager-849f85686b-79g95\" (UID: \"ed50ed7c-3871-439c-9d3e-e07b609fefaa\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:03:53.980717 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.980695 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ed50ed7c-3871-439c-9d3e-e07b609fefaa-manager-config\") pod \"lws-controller-manager-849f85686b-79g95\" (UID: \"ed50ed7c-3871-439c-9d3e-e07b609fefaa\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:03:53.982747 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.982723 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed50ed7c-3871-439c-9d3e-e07b609fefaa-cert\") pod \"lws-controller-manager-849f85686b-79g95\" (UID: \"ed50ed7c-3871-439c-9d3e-e07b609fefaa\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:03:53.983072 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.983051 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed50ed7c-3871-439c-9d3e-e07b609fefaa-metrics-cert\") pod \"lws-controller-manager-849f85686b-79g95\" (UID: \"ed50ed7c-3871-439c-9d3e-e07b609fefaa\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:03:53.991103 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:53.991077 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqk6r\" (UniqueName: \"kubernetes.io/projected/ed50ed7c-3871-439c-9d3e-e07b609fefaa-kube-api-access-vqk6r\") pod \"lws-controller-manager-849f85686b-79g95\" (UID: \"ed50ed7c-3871-439c-9d3e-e07b609fefaa\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:03:54.028097 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:54.028064 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:03:54.221820 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:54.221795 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh"] Sep 06 14:03:54.312475 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:54.312443 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-849f85686b-79g95"] Sep 06 14:03:54.315339 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:03:54.315315 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded50ed7c_3871_439c_9d3e_e07b609fefaa.slice/crio-7e6f554ccbdee9107054b5ef415314cb9b10ed7988bd374240e970166e0c7390 WatchSource:0}: Error finding container 7e6f554ccbdee9107054b5ef415314cb9b10ed7988bd374240e970166e0c7390: Status 404 returned error can't find the container with id 7e6f554ccbdee9107054b5ef415314cb9b10ed7988bd374240e970166e0c7390 Sep 06 14:03:55.010658 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:55.010615 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" event={"ID":"ed50ed7c-3871-439c-9d3e-e07b609fefaa","Type":"ContainerStarted","Data":"7e6f554ccbdee9107054b5ef415314cb9b10ed7988bd374240e970166e0c7390"} Sep 06 14:03:55.012194 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:55.012157 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" event={"ID":"42eb9483-2a9e-4816-957c-5bfedefda1d2","Type":"ContainerStarted","Data":"769a010fc5fdf7983868d955fc66d5405160ea5ed579940c27db69de06953dcf"} Sep 06 14:03:56.069647 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:56.069551 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:56.161132 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:56.161091 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:56.389285 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:56.389203 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:56.750283 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:56.750192 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:57.355545 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:57.355500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:57.457012 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:57.456977 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:57.557129 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:57.557097 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:03:58.197329 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:58.197298 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:03:58.197401 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:58.197374 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:03:59.040087 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:59.040053 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" event={"ID":"42eb9483-2a9e-4816-957c-5bfedefda1d2","Type":"ContainerStarted","Data":"bccbf17dd2a4274bef536a8e430321cc46035a7a1ad7782d16be69f5cbcffe5d"} Sep 06 14:03:59.040544 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:59.040107 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:59.041701 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:59.041665 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" event={"ID":"ed50ed7c-3871-439c-9d3e-e07b609fefaa","Type":"ContainerStarted","Data":"36473a089de750b7587fc60ba2a1cbf2b6dc306b83ad4ad63aa749ae63e4e911"} Sep 06 14:03:59.041824 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:59.041739 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:03:59.042140 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:59.042125 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" Sep 06 14:03:59.059222 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:59.059178 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wfwsh" podStartSLOduration=2.091704174 podStartE2EDuration="6.059166237s" totalImagesPullingTime="3.967462063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:03:59.057718949 +0000 UTC m=+370.513207812" watchObservedRunningTime="2026-09-06 14:03:59.059166237 +0000 UTC m=+370.514655098" Sep 06 14:03:59.072606 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:03:59.072565 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" podStartSLOduration=2.19500084 podStartE2EDuration="6.0725481s" totalImagesPullingTime="3.87754726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:03:59.071266683 +0000 UTC m=+370.526755545" watchObservedRunningTime="2026-09-06 14:03:59.0725481 +0000 UTC m=+370.528036964" Sep 06 14:04:01.995538 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:01.995505 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-cwsj8" Sep 06 14:04:06.104894 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:06.104860 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:04:06.152407 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:06.152377 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:04:06.247815 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:06.247783 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:04:06.850763 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:06.850732 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:04:10.048540 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:10.048512 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-849f85686b-79g95" Sep 06 14:04:17.312272 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:17.312234 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:04:17.366965 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:17.366932 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:04:17.414522 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:17.414479 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:04:47.516163 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:47.516120 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:04:47.618891 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:47.618848 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:04:57.545143 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:57.545104 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:04:57.598732 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:57.598705 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:04:59.115490 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:59.115459 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:04:59.504923 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:59.504806 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:04:59.606279 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:59.606239 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:04:59.919418 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:04:59.919383 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:00.720697 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:00.720665 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:03.997526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:03.997484 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:04.145014 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:04.144984 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:06.123828 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:06.123791 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:06.366666 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:06.366636 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:07.765633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:07.765595 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:07.930035 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:07.930000 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:08.530496 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:08.530461 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:09.957109 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:09.957070 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:10.060357 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:10.060315 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:11.347958 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:11.347864 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:11.778208 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:11.778177 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:14.320754 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:14.320715 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:14.373639 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:14.373601 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:14.720536 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:14.720457 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:14.768337 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:14.768305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:15.121514 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:15.121478 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:15.180305 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:15.180272 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:20.795196 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:20.795161 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:20.901478 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:20.901430 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:25.557748 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:25.557710 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:25.678703 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:25.678672 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:30.341658 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:30.341622 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:30.439930 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:30.439882 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:35.341038 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:35.341004 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:35.441005 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:35.440975 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:35.813638 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:35.813604 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:35.914271 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:35.914235 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:36.138157 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:36.138079 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:36.188677 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:36.188647 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:41.965466 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:41.965431 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:42.065136 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:42.065107 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:42.597208 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:42.597175 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:42.658184 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:42.658151 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:47.346319 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:47.346285 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:47.447056 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:47.447023 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:49.228117 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:49.228083 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:49.278982 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:49.278952 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:50.914992 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:50.914961 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl"] Sep 06 14:05:50.918437 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:50.918415 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:50.920601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:50.920581 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gb2xd\"" Sep 06 14:05:50.929336 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:50.929312 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl"] Sep 06 14:05:50.943505 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:50.943482 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:50.969715 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:50.969692 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0479d3c1-0872-435a-8133-8e570afa1d85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:50.969715 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:50.969723 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0479d3c1-0872-435a-8133-8e570afa1d85-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:50.969877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:50.969752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkv7c\" (UniqueName: \"kubernetes.io/projected/0479d3c1-0872-435a-8133-8e570afa1d85-kube-api-access-qkv7c\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:50.969877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:50.969838 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0479d3c1-0872-435a-8133-8e570afa1d85-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:50.969877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:50.969865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0479d3c1-0872-435a-8133-8e570afa1d85-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:50.970029 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:50.969881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0479d3c1-0872-435a-8133-8e570afa1d85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:50.970029 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:50.969900 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0479d3c1-0872-435a-8133-8e570afa1d85-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:50.970029 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:50.969933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0479d3c1-0872-435a-8133-8e570afa1d85-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:50.970029 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:50.969983 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0479d3c1-0872-435a-8133-8e570afa1d85-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.070419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.070390 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0479d3c1-0872-435a-8133-8e570afa1d85-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.070584 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.070469 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0479d3c1-0872-435a-8133-8e570afa1d85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.070584 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.070490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0479d3c1-0872-435a-8133-8e570afa1d85-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.070584 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.070515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkv7c\" (UniqueName: \"kubernetes.io/projected/0479d3c1-0872-435a-8133-8e570afa1d85-kube-api-access-qkv7c\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.070584 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.070553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0479d3c1-0872-435a-8133-8e570afa1d85-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.070584 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.070569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0479d3c1-0872-435a-8133-8e570afa1d85-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.070832 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.070686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0479d3c1-0872-435a-8133-8e570afa1d85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.070832 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.070723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0479d3c1-0872-435a-8133-8e570afa1d85-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.070832 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.070753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0479d3c1-0872-435a-8133-8e570afa1d85-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.071021 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.070970 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0479d3c1-0872-435a-8133-8e570afa1d85-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.071078 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.071040 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0479d3c1-0872-435a-8133-8e570afa1d85-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.071229 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.071212 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0479d3c1-0872-435a-8133-8e570afa1d85-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.071285 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.071252 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0479d3c1-0872-435a-8133-8e570afa1d85-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.071397 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.071366 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0479d3c1-0872-435a-8133-8e570afa1d85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.072768 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.072747 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0479d3c1-0872-435a-8133-8e570afa1d85-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.073083 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.073066 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0479d3c1-0872-435a-8133-8e570afa1d85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.077190 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.077163 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0479d3c1-0872-435a-8133-8e570afa1d85-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.077313 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.077259 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkv7c\" (UniqueName: \"kubernetes.io/projected/0479d3c1-0872-435a-8133-8e570afa1d85-kube-api-access-qkv7c\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9m4jl\" (UID: \"0479d3c1-0872-435a-8133-8e570afa1d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.230163 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.230088 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:51.288960 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.288933 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:51.504060 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.504038 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl"] Sep 06 14:05:51.506291 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:05:51.506261 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0479d3c1_0872_435a_8133_8e570afa1d85.slice/crio-bbac4d63687ec2c68d9fa75b24994b2d8cb47680b97d3e3664eabb8cf7b9a60f WatchSource:0}: Error finding container bbac4d63687ec2c68d9fa75b24994b2d8cb47680b97d3e3664eabb8cf7b9a60f: Status 404 returned error can't find the container with id bbac4d63687ec2c68d9fa75b24994b2d8cb47680b97d3e3664eabb8cf7b9a60f Sep 06 14:05:51.552921 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:51.552884 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" event={"ID":"0479d3c1-0872-435a-8133-8e570afa1d85","Type":"ContainerStarted","Data":"bbac4d63687ec2c68d9fa75b24994b2d8cb47680b97d3e3664eabb8cf7b9a60f"} Sep 06 14:05:52.145705 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:52.145674 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:52.245633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:52.245593 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:54.124561 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:54.124529 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:05:54.124805 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:54.124598 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:05:54.124805 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:54.124627 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:05:54.569662 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:54.569626 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" event={"ID":"0479d3c1-0872-435a-8133-8e570afa1d85","Type":"ContainerStarted","Data":"b55c73f5a32f5328bb802d13554b695ac22e6deb5cbdb07995eae18932ab5797"} Sep 06 14:05:54.588274 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:54.588214 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" podStartSLOduration=1.972231209 podStartE2EDuration="4.588193651s" totalImagesPullingTime="2.615962442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:05:54.585903268 +0000 UTC m=+486.041392131" watchObservedRunningTime="2026-09-06 14:05:54.588193651 +0000 UTC m=+486.043682515" Sep 06 14:05:55.230950 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:55.230898 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:55.236109 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:55.236085 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:55.519821 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:55.519789 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:55.574774 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:55.574742 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:55.575688 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:55.575668 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:05:55.629369 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:55.629339 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:55.680309 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:55.680278 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:55.921482 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:55.921409 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:05:55.925118 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:55.925102 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf" Sep 06 14:05:55.927827 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:55.927799 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 06 14:05:55.928597 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:55.928575 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 06 14:05:55.928699 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:55.928601 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 06 14:05:55.928845 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:55.928830 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6bpnv\"" Sep 06 14:05:55.938978 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:55.938960 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:05:55.985938 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:55.985898 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:05:56.006655 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:56.006633 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/163b006d-dcb7-4c0b-acfd-579d88d2b6fb-config-file\") pod \"limitador-limitador-85d9db4d5d-7jbpf\" (UID: \"163b006d-dcb7-4c0b-acfd-579d88d2b6fb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf" Sep 06 14:05:56.006753 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:56.006658 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zcrm\" (UniqueName: \"kubernetes.io/projected/163b006d-dcb7-4c0b-acfd-579d88d2b6fb-kube-api-access-5zcrm\") pod \"limitador-limitador-85d9db4d5d-7jbpf\" (UID: \"163b006d-dcb7-4c0b-acfd-579d88d2b6fb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf" Sep 06 14:05:56.107571 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:56.107537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/163b006d-dcb7-4c0b-acfd-579d88d2b6fb-config-file\") pod \"limitador-limitador-85d9db4d5d-7jbpf\" (UID: \"163b006d-dcb7-4c0b-acfd-579d88d2b6fb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf" Sep 06 14:05:56.107571 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:56.107573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zcrm\" (UniqueName: \"kubernetes.io/projected/163b006d-dcb7-4c0b-acfd-579d88d2b6fb-kube-api-access-5zcrm\") pod \"limitador-limitador-85d9db4d5d-7jbpf\" (UID: \"163b006d-dcb7-4c0b-acfd-579d88d2b6fb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf" Sep 06 14:05:56.108277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:56.108258 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/163b006d-dcb7-4c0b-acfd-579d88d2b6fb-config-file\") pod \"limitador-limitador-85d9db4d5d-7jbpf\" (UID: \"163b006d-dcb7-4c0b-acfd-579d88d2b6fb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf" Sep 06 14:05:56.114603 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:56.114583 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zcrm\" (UniqueName: \"kubernetes.io/projected/163b006d-dcb7-4c0b-acfd-579d88d2b6fb-kube-api-access-5zcrm\") pod \"limitador-limitador-85d9db4d5d-7jbpf\" (UID: \"163b006d-dcb7-4c0b-acfd-579d88d2b6fb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf" Sep 06 14:05:56.234943 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:56.234843 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf" Sep 06 14:05:56.433877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:56.433842 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:56.499808 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:56.499714 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:05:56.510421 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:56.510307 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:05:56.512770 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:05:56.512743 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163b006d_dcb7_4c0b_acfd_579d88d2b6fb.slice/crio-f4296aadec795b42b3798f485d18af954c30e0d4516c896ccb42fb6a08c408e9 WatchSource:0}: Error finding container f4296aadec795b42b3798f485d18af954c30e0d4516c896ccb42fb6a08c408e9: Status 404 returned error can't find the container with id f4296aadec795b42b3798f485d18af954c30e0d4516c896ccb42fb6a08c408e9 Sep 06 14:05:56.579993 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:56.579961 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf" event={"ID":"163b006d-dcb7-4c0b-acfd-579d88d2b6fb","Type":"ContainerStarted","Data":"f4296aadec795b42b3798f485d18af954c30e0d4516c896ccb42fb6a08c408e9"} Sep 06 14:05:58.594969 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:58.594939 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf" event={"ID":"163b006d-dcb7-4c0b-acfd-579d88d2b6fb","Type":"ContainerStarted","Data":"19f691eb60ec08f2dee40d758e5ed6412c196378bf034547dfe773d3f3e5eaee"} Sep 06 14:05:58.595324 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:58.595085 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf" Sep 06 14:05:58.608822 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:05:58.608783 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf" podStartSLOduration=1.665669912 podStartE2EDuration="3.608751372s" totalImagesPullingTime="1.94308146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:05:58.607715714 +0000 UTC m=+490.063204577" watchObservedRunningTime="2026-09-06 14:05:58.608751372 +0000 UTC m=+490.064240234" Sep 06 14:06:09.600665 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:09.600634 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf" Sep 06 14:06:11.594656 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:11.594622 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:11.693605 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:11.693568 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:12.041212 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:12.041182 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:21.963394 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:21.963334 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:22.016678 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:22.016649 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:24.269858 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:24.269825 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:24.613542 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:24.613456 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:30.524740 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:30.524705 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:30.623136 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:30.623102 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:30.675833 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:30.675800 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:30.741105 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:30.741077 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:30.841838 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:30.841753 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:34.206396 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:34.206361 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:34.306025 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:34.305995 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:58.025869 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:58.025832 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:58.513279 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:58.513177 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:06:58.561303 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:06:58.561273 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:07:06.756722 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:06.756673 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:07:06.858864 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:06.858830 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:07:08.330879 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:08.330847 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:07:08.409332 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:08.409300 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:07:08.535823 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:08.535792 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:07:08.636410 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:08.636315 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:07:08.874575 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:08.874532 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:07:08.974285 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:08.974196 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:07:09.022032 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:09.021997 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-shz4m"] Sep 06 14:07:09.025703 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:09.025687 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-shz4m" Sep 06 14:07:09.027943 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:09.027916 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pjcd6\"" Sep 06 14:07:09.028077 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:09.027957 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 06 14:07:09.038081 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:09.038057 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-shz4m"] Sep 06 14:07:09.045980 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:09.045955 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:07:09.121113 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:09.121059 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434bf1f2-d6df-4fc0-a8ae-49216214dd8c-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-shz4m\" (UID: \"434bf1f2-d6df-4fc0-a8ae-49216214dd8c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-shz4m" Sep 06 14:07:09.121296 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:09.121198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc2k7\" (UniqueName: \"kubernetes.io/projected/434bf1f2-d6df-4fc0-a8ae-49216214dd8c-kube-api-access-nc2k7\") pod \"payload-processing-5b54b7fbb7-shz4m\" (UID: \"434bf1f2-d6df-4fc0-a8ae-49216214dd8c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-shz4m" Sep 06 14:07:09.222481 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:09.222447 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434bf1f2-d6df-4fc0-a8ae-49216214dd8c-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-shz4m\" (UID: \"434bf1f2-d6df-4fc0-a8ae-49216214dd8c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-shz4m" Sep 06 14:07:09.222657 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:09.222628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc2k7\" (UniqueName: \"kubernetes.io/projected/434bf1f2-d6df-4fc0-a8ae-49216214dd8c-kube-api-access-nc2k7\") pod \"payload-processing-5b54b7fbb7-shz4m\" (UID: \"434bf1f2-d6df-4fc0-a8ae-49216214dd8c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-shz4m" Sep 06 14:07:09.223212 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:09.223189 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434bf1f2-d6df-4fc0-a8ae-49216214dd8c-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-shz4m\" (UID: \"434bf1f2-d6df-4fc0-a8ae-49216214dd8c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-shz4m" Sep 06 14:07:09.229800 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:09.229744 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc2k7\" (UniqueName: \"kubernetes.io/projected/434bf1f2-d6df-4fc0-a8ae-49216214dd8c-kube-api-access-nc2k7\") pod \"payload-processing-5b54b7fbb7-shz4m\" (UID: \"434bf1f2-d6df-4fc0-a8ae-49216214dd8c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-shz4m" Sep 06 14:07:09.340471 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:09.340432 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-shz4m" Sep 06 14:07:09.608230 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:09.608199 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-shz4m"] Sep 06 14:07:09.611131 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:07:09.611100 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434bf1f2_d6df_4fc0_a8ae_49216214dd8c.slice/crio-ec272ae782073efe2b827b22e3962007b5be3a00d659184e568e6581c2dcf428 WatchSource:0}: Error finding container ec272ae782073efe2b827b22e3962007b5be3a00d659184e568e6581c2dcf428: Status 404 returned error can't find the container with id ec272ae782073efe2b827b22e3962007b5be3a00d659184e568e6581c2dcf428 Sep 06 14:07:09.925746 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:09.925637 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-shz4m" event={"ID":"434bf1f2-d6df-4fc0-a8ae-49216214dd8c","Type":"ContainerStarted","Data":"ec272ae782073efe2b827b22e3962007b5be3a00d659184e568e6581c2dcf428"} Sep 06 14:07:11.959015 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:11.958981 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-shz4m" event={"ID":"434bf1f2-d6df-4fc0-a8ae-49216214dd8c","Type":"ContainerStarted","Data":"0e07b93b267653d9756e02b9aa837186a271393abc2a23d6e8d8fa290f189342"} Sep 06 14:07:11.959504 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:11.959444 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-shz4m" Sep 06 14:07:11.974683 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:11.974634 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-shz4m" podStartSLOduration=1.190410106 podStartE2EDuration="2.974619147s" totalImagesPullingTime="1.784209041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:07:11.973149066 +0000 UTC m=+563.428637966" watchObservedRunningTime="2026-09-06 14:07:11.974619147 +0000 UTC m=+563.430108009" Sep 06 14:07:17.989322 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:17.989291 2577 generic.go:356] "Generic (PLEG): container finished" podID="0479d3c1-0872-435a-8133-8e570afa1d85" containerID="b55c73f5a32f5328bb802d13554b695ac22e6deb5cbdb07995eae18932ab5797" exitCode=0 Sep 06 14:07:17.989708 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:17.989361 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" event={"ID":"0479d3c1-0872-435a-8133-8e570afa1d85","Type":"ContainerDied","Data":"b55c73f5a32f5328bb802d13554b695ac22e6deb5cbdb07995eae18932ab5797"} Sep 06 14:07:17.989775 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:17.989735 2577 scope.go:117] "RemoveContainer" containerID="b55c73f5a32f5328bb802d13554b695ac22e6deb5cbdb07995eae18932ab5797" Sep 06 14:07:17.990762 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:17.990737 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:07:17.990823 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:17.990809 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:07:17.990860 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:17.990842 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:07:18.249802 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:18.249706 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:07:18.350367 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:18.350329 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:07:18.996940 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:18.996889 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" event={"ID":"0479d3c1-0872-435a-8133-8e570afa1d85","Type":"ContainerStarted","Data":"3802c39aad04389e35e8385d2c0284c2898dd6eda043cfd9aa7b0b3c611626a6"} Sep 06 14:07:19.239931 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:19.239744 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:07:19.239931 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:19.239815 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9m4jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Sep 06 14:07:19.239931 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:19.239864 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" podUID="0479d3c1-0872-435a-8133-8e570afa1d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Sep 06 14:07:20.231005 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:20.230966 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9m4jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Sep 06 14:07:20.231471 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:20.231044 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" podUID="0479d3c1-0872-435a-8133-8e570afa1d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Sep 06 14:07:21.230743 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:21.230705 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9m4jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Sep 06 14:07:21.231074 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:21.231034 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" podUID="0479d3c1-0872-435a-8133-8e570afa1d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Sep 06 14:07:22.231458 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:22.231416 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9m4jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Sep 06 14:07:22.231958 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:22.231492 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" podUID="0479d3c1-0872-435a-8133-8e570afa1d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Sep 06 14:07:22.966082 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:22.966052 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-shz4m" Sep 06 14:07:23.230816 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:23.230721 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9m4jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Sep 06 14:07:23.231091 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:23.231061 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" podUID="0479d3c1-0872-435a-8133-8e570afa1d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Sep 06 14:07:24.235401 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:24.235371 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:07:24.235827 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:24.235630 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:07:24.236255 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:24.236240 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9m4jl" Sep 06 14:07:24.862577 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:24.862526 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:07:24.911921 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:24.911877 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:07:32.828446 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:32.828413 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:07:33.392893 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:07:33.392862 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:08:01.098222 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:01.098183 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:08:01.690132 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:01.690087 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:08:07.120542 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:07.120511 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:08:08.087821 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:08.087793 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:08:24.527048 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:24.526971 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:08:25.794649 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:25.794616 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:08:35.292145 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.292108 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2"] Sep 06 14:08:35.294772 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.294752 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.297004 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.296982 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l4dz5\"" Sep 06 14:08:35.298036 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.297973 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 06 14:08:35.298220 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.298204 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 06 14:08:35.298361 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.298341 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 06 14:08:35.304042 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.304020 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2"] Sep 06 14:08:35.386635 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.386603 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:08:35.387702 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.387679 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0766686-d24e-46e6-b3a5-0f25393d0fc4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.387798 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.387718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0766686-d24e-46e6-b3a5-0f25393d0fc4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.387798 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.387741 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0766686-d24e-46e6-b3a5-0f25393d0fc4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.387798 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.387793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwvbj\" (UniqueName: \"kubernetes.io/projected/b0766686-d24e-46e6-b3a5-0f25393d0fc4-kube-api-access-fwvbj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.387933 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.387811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0766686-d24e-46e6-b3a5-0f25393d0fc4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.387933 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.387830 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0766686-d24e-46e6-b3a5-0f25393d0fc4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.488465 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.488432 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0766686-d24e-46e6-b3a5-0f25393d0fc4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.488653 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.488484 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0766686-d24e-46e6-b3a5-0f25393d0fc4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.488653 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.488503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0766686-d24e-46e6-b3a5-0f25393d0fc4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.488653 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.488551 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwvbj\" (UniqueName: \"kubernetes.io/projected/b0766686-d24e-46e6-b3a5-0f25393d0fc4-kube-api-access-fwvbj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.488829 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.488660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0766686-d24e-46e6-b3a5-0f25393d0fc4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.488829 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.488689 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0766686-d24e-46e6-b3a5-0f25393d0fc4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.489240 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.489044 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0766686-d24e-46e6-b3a5-0f25393d0fc4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.489240 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.489085 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0766686-d24e-46e6-b3a5-0f25393d0fc4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.489240 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.489186 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0766686-d24e-46e6-b3a5-0f25393d0fc4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.490872 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.490849 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0766686-d24e-46e6-b3a5-0f25393d0fc4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.491662 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.491645 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0766686-d24e-46e6-b3a5-0f25393d0fc4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.495093 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.495072 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwvbj\" (UniqueName: \"kubernetes.io/projected/b0766686-d24e-46e6-b3a5-0f25393d0fc4-kube-api-access-fwvbj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2\" (UID: \"b0766686-d24e-46e6-b3a5-0f25393d0fc4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.605414 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.605356 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:35.871613 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.871589 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2"] Sep 06 14:08:35.873505 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:08:35.873479 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0766686_d24e_46e6_b3a5_0f25393d0fc4.slice/crio-6ccee7cbe174d7663dfa098486094e455f1059f7f09c41dc4baef405012907d0 WatchSource:0}: Error finding container 6ccee7cbe174d7663dfa098486094e455f1059f7f09c41dc4baef405012907d0: Status 404 returned error can't find the container with id 6ccee7cbe174d7663dfa098486094e455f1059f7f09c41dc4baef405012907d0 Sep 06 14:08:35.874830 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:35.874813 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 14:08:36.366138 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:36.366097 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" event={"ID":"b0766686-d24e-46e6-b3a5-0f25393d0fc4","Type":"ContainerStarted","Data":"6ccee7cbe174d7663dfa098486094e455f1059f7f09c41dc4baef405012907d0"} Sep 06 14:08:36.789508 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:36.789482 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:08:37.373436 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:37.373398 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" event={"ID":"b0766686-d24e-46e6-b3a5-0f25393d0fc4","Type":"ContainerStarted","Data":"ccfd9f3e1dcc4387025078427a3ee04dac8c4222709cc0cfd74a515049d5cf84"} Sep 06 14:08:37.390151 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:37.390101 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" podStartSLOduration=1.48274507 podStartE2EDuration="2.390085873s" totalImagesPullingTime="907.340803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:08:37.388161641 +0000 UTC m=+648.843650520" watchObservedRunningTime="2026-09-06 14:08:37.390085873 +0000 UTC m=+648.845574734" Sep 06 14:08:40.501246 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:40.501208 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:08:40.602407 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:40.602377 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:08:45.605780 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:45.605749 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:45.606249 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:45.605826 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:45.618123 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:45.618103 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:45.830445 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:45.830406 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:08:45.931251 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:45.931180 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:08:46.426923 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:46.426887 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2" Sep 06 14:08:56.992925 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:56.992859 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88"] Sep 06 14:08:56.997934 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:56.997901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.000230 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.000212 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 06 14:08:57.003884 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.003860 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88"] Sep 06 14:08:57.028543 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.028515 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:08:57.069992 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.069966 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02261a3a-c803-4c8d-a93d-db27f1543676-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.070105 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.070005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02261a3a-c803-4c8d-a93d-db27f1543676-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.070169 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.070099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqlhz\" (UniqueName: \"kubernetes.io/projected/02261a3a-c803-4c8d-a93d-db27f1543676-kube-api-access-fqlhz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.070169 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.070162 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02261a3a-c803-4c8d-a93d-db27f1543676-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.070280 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.070200 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02261a3a-c803-4c8d-a93d-db27f1543676-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.070335 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.070311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02261a3a-c803-4c8d-a93d-db27f1543676-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.171692 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.171665 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqlhz\" (UniqueName: \"kubernetes.io/projected/02261a3a-c803-4c8d-a93d-db27f1543676-kube-api-access-fqlhz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.171841 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.171713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02261a3a-c803-4c8d-a93d-db27f1543676-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.171841 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.171744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02261a3a-c803-4c8d-a93d-db27f1543676-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.171841 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.171781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02261a3a-c803-4c8d-a93d-db27f1543676-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.171841 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.171801 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02261a3a-c803-4c8d-a93d-db27f1543676-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.171841 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.171824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02261a3a-c803-4c8d-a93d-db27f1543676-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.172184 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.172163 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02261a3a-c803-4c8d-a93d-db27f1543676-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.172251 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.172191 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02261a3a-c803-4c8d-a93d-db27f1543676-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.172315 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.172292 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02261a3a-c803-4c8d-a93d-db27f1543676-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.174156 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.174142 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02261a3a-c803-4c8d-a93d-db27f1543676-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.174344 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.174327 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02261a3a-c803-4c8d-a93d-db27f1543676-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.178388 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.178365 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqlhz\" (UniqueName: \"kubernetes.io/projected/02261a3a-c803-4c8d-a93d-db27f1543676-kube-api-access-fqlhz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpw88\" (UID: \"02261a3a-c803-4c8d-a93d-db27f1543676\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.312021 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.311983 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:08:57.580389 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:57.580366 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88"] Sep 06 14:08:57.582128 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:08:57.582101 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02261a3a_c803_4c8d_a93d_db27f1543676.slice/crio-d72abff6a7aab216e6d578a9ba62c947e412405824574afb3f8d0f0460d6be50 WatchSource:0}: Error finding container d72abff6a7aab216e6d578a9ba62c947e412405824574afb3f8d0f0460d6be50: Status 404 returned error can't find the container with id d72abff6a7aab216e6d578a9ba62c947e412405824574afb3f8d0f0460d6be50 Sep 06 14:08:58.473922 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:58.473874 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" event={"ID":"02261a3a-c803-4c8d-a93d-db27f1543676","Type":"ContainerStarted","Data":"60e80701cdb14892353459550756e959f80c88c8f3fc28e942ba22624eec8fe2"} Sep 06 14:08:58.474391 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:58.473938 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" event={"ID":"02261a3a-c803-4c8d-a93d-db27f1543676","Type":"ContainerStarted","Data":"d72abff6a7aab216e6d578a9ba62c947e412405824574afb3f8d0f0460d6be50"} Sep 06 14:08:58.490612 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:58.490564 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" podStartSLOduration=2.314145034 podStartE2EDuration="2.4905402s" totalImagesPullingTime="176.395166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:08:58.488490366 +0000 UTC m=+669.943979235" watchObservedRunningTime="2026-09-06 14:08:58.4905402 +0000 UTC m=+669.946029063" Sep 06 14:08:58.789446 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:08:58.789412 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:09:07.312656 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:09:07.312620 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:09:07.313068 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:09:07.312681 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:09:07.324945 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:09:07.324924 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:09:07.525026 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:09:07.524993 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpw88" Sep 06 14:09:11.611557 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:09:11.611514 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:09:13.894404 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:09:13.894375 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:09:19.950334 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:09:19.950294 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:08.900980 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:08.900937 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:09.002516 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:09.002474 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:09.941462 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:09.941427 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:10.040324 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:10.040273 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:25.333501 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:25.333450 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:25.447695 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:25.447642 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:25.651544 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:25.651460 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:26.221364 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:26.221321 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:26.658310 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:26.658270 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:26.766792 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:26.766759 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:27.109253 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:27.109218 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:28.494442 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:28.494404 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:28.593339 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:28.593304 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:29.101735 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:29.101702 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:10:29.670945 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:29.670898 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:29.770194 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:29.770160 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:30.126595 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.126555 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:30.221699 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.221672 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:30.225452 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.225425 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht"] Sep 06 14:10:30.229053 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.229033 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" Sep 06 14:10:30.231405 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.231383 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1dfd6b-dockercfg-jxkkb\"" Sep 06 14:10:30.231493 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.231421 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1dfd6b\"" Sep 06 14:10:30.241163 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.241138 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht"] Sep 06 14:10:30.252041 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.252019 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt"] Sep 06 14:10:30.255945 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.255926 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" Sep 06 14:10:30.266029 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.266007 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt"] Sep 06 14:10:30.306384 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.306357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bcp\" (UniqueName: \"kubernetes.io/projected/0191f47a-0637-42c6-abd3-eb7f02ece7c3-kube-api-access-n4bcp\") pod \"payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht\" (UID: \"0191f47a-0637-42c6-abd3-eb7f02ece7c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" Sep 06 14:10:30.306499 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.306431 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0191f47a-0637-42c6-abd3-eb7f02ece7c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht\" (UID: \"0191f47a-0637-42c6-abd3-eb7f02ece7c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" Sep 06 14:10:30.407752 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.407665 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bcp\" (UniqueName: \"kubernetes.io/projected/0191f47a-0637-42c6-abd3-eb7f02ece7c3-kube-api-access-n4bcp\") pod \"payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht\" (UID: \"0191f47a-0637-42c6-abd3-eb7f02ece7c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" Sep 06 14:10:30.407752 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.407733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbcb2df1-4343-4f25-9d05-a8c75df4f2eb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt\" (UID: \"dbcb2df1-4343-4f25-9d05-a8c75df4f2eb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" Sep 06 14:10:30.407958 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.407809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqjsq\" (UniqueName: \"kubernetes.io/projected/dbcb2df1-4343-4f25-9d05-a8c75df4f2eb-kube-api-access-nqjsq\") pod \"payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt\" (UID: \"dbcb2df1-4343-4f25-9d05-a8c75df4f2eb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" Sep 06 14:10:30.407958 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.407869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0191f47a-0637-42c6-abd3-eb7f02ece7c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht\" (UID: \"0191f47a-0637-42c6-abd3-eb7f02ece7c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" Sep 06 14:10:30.408522 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.408503 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0191f47a-0637-42c6-abd3-eb7f02ece7c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht\" (UID: \"0191f47a-0637-42c6-abd3-eb7f02ece7c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" Sep 06 14:10:30.414900 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.414870 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bcp\" (UniqueName: \"kubernetes.io/projected/0191f47a-0637-42c6-abd3-eb7f02ece7c3-kube-api-access-n4bcp\") pod \"payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht\" (UID: \"0191f47a-0637-42c6-abd3-eb7f02ece7c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" Sep 06 14:10:30.509330 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.509295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbcb2df1-4343-4f25-9d05-a8c75df4f2eb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt\" (UID: \"dbcb2df1-4343-4f25-9d05-a8c75df4f2eb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" Sep 06 14:10:30.509478 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.509343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqjsq\" (UniqueName: \"kubernetes.io/projected/dbcb2df1-4343-4f25-9d05-a8c75df4f2eb-kube-api-access-nqjsq\") pod \"payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt\" (UID: \"dbcb2df1-4343-4f25-9d05-a8c75df4f2eb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" Sep 06 14:10:30.510333 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.510308 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbcb2df1-4343-4f25-9d05-a8c75df4f2eb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt\" (UID: \"dbcb2df1-4343-4f25-9d05-a8c75df4f2eb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" Sep 06 14:10:30.516837 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.516806 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqjsq\" (UniqueName: \"kubernetes.io/projected/dbcb2df1-4343-4f25-9d05-a8c75df4f2eb-kube-api-access-nqjsq\") pod \"payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt\" (UID: \"dbcb2df1-4343-4f25-9d05-a8c75df4f2eb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" Sep 06 14:10:30.540563 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.540545 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" Sep 06 14:10:30.566478 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.566451 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" Sep 06 14:10:30.812852 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.812826 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht"] Sep 06 14:10:30.814679 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:10:30.814652 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0191f47a_0637_42c6_abd3_eb7f02ece7c3.slice/crio-dfcad1e2bff738e265981be12efde5f89b0bab14fdada89a252eb4363e5475dd WatchSource:0}: Error finding container dfcad1e2bff738e265981be12efde5f89b0bab14fdada89a252eb4363e5475dd: Status 404 returned error can't find the container with id dfcad1e2bff738e265981be12efde5f89b0bab14fdada89a252eb4363e5475dd Sep 06 14:10:30.853940 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.853893 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt"] Sep 06 14:10:30.854970 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:10:30.854942 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcb2df1_4343_4f25_9d05_a8c75df4f2eb.slice/crio-b82ec21cca10ed289302b61203d03fd078c01b58798dece38b306707afdeb554 WatchSource:0}: Error finding container b82ec21cca10ed289302b61203d03fd078c01b58798dece38b306707afdeb554: Status 404 returned error can't find the container with id b82ec21cca10ed289302b61203d03fd078c01b58798dece38b306707afdeb554 Sep 06 14:10:30.920076 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.920042 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" event={"ID":"dbcb2df1-4343-4f25-9d05-a8c75df4f2eb","Type":"ContainerStarted","Data":"e0d835de59f65c3cd0a9f843725e363eecabab0cbaae34510f3fb65e7bb53db6"} Sep 06 14:10:30.920257 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.920105 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" event={"ID":"dbcb2df1-4343-4f25-9d05-a8c75df4f2eb","Type":"ContainerStarted","Data":"b82ec21cca10ed289302b61203d03fd078c01b58798dece38b306707afdeb554"} Sep 06 14:10:30.920257 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.920156 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" Sep 06 14:10:30.921651 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.921630 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" event={"ID":"0191f47a-0637-42c6-abd3-eb7f02ece7c3","Type":"ContainerStarted","Data":"4f88b898df81ba4d1668a9122f966013a3f054a66ee3f741f710bf53b845de02"} Sep 06 14:10:30.921779 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.921670 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" event={"ID":"0191f47a-0637-42c6-abd3-eb7f02ece7c3","Type":"ContainerStarted","Data":"dfcad1e2bff738e265981be12efde5f89b0bab14fdada89a252eb4363e5475dd"} Sep 06 14:10:30.921779 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.921698 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" Sep 06 14:10:30.936173 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.936115 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" podStartSLOduration=0.936093517 podStartE2EDuration="936.093517ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:30.933235582 +0000 UTC m=+762.388724447" watchObservedRunningTime="2026-09-06 14:10:30.936093517 +0000 UTC m=+762.391582379" Sep 06 14:10:30.950807 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:30.950740 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" podStartSLOduration=0.950719987 podStartE2EDuration="950.719987ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:30.947799774 +0000 UTC m=+762.403288632" watchObservedRunningTime="2026-09-06 14:10:30.950719987 +0000 UTC m=+762.406208850" Sep 06 14:10:32.989435 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:32.989403 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:33.041473 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:33.041443 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:33.208717 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:33.208689 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:33.508068 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:33.508031 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:33.702865 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:33.702834 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:35.276666 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:35.276630 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:35.560969 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:35.560868 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:35.964734 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:35.964637 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:36.061782 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:36.061753 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:37.126519 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.126484 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:37.234012 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.233964 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:37.326208 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.326167 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:37.596485 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.596453 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:37.693386 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.693340 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:37.697237 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.697210 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf"] Sep 06 14:10:37.701314 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.701297 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" Sep 06 14:10:37.703410 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.703378 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-84753aa3-dockercfg-j4zlh\"" Sep 06 14:10:37.703516 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.703408 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-84753aa3\"" Sep 06 14:10:37.710389 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.710366 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf"] Sep 06 14:10:37.734691 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.734663 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx"] Sep 06 14:10:37.738792 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.738772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" Sep 06 14:10:37.745510 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.745488 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx"] Sep 06 14:10:37.883251 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.883155 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsgtl\" (UniqueName: \"kubernetes.io/projected/b149c687-9e38-46a3-90d7-b9b6201b2700-kube-api-access-lsgtl\") pod \"payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx\" (UID: \"b149c687-9e38-46a3-90d7-b9b6201b2700\") " pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" Sep 06 14:10:37.883251 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.883233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e093cad-6f89-4c15-8391-928c3dc25d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf\" (UID: \"5e093cad-6f89-4c15-8391-928c3dc25d79\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" Sep 06 14:10:37.883493 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.883269 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b149c687-9e38-46a3-90d7-b9b6201b2700-plugins-config-volume\") pod \"payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx\" (UID: \"b149c687-9e38-46a3-90d7-b9b6201b2700\") " pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" Sep 06 14:10:37.883493 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.883292 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtf8\" (UniqueName: \"kubernetes.io/projected/5e093cad-6f89-4c15-8391-928c3dc25d79-kube-api-access-9mtf8\") pod \"payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf\" (UID: \"5e093cad-6f89-4c15-8391-928c3dc25d79\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" Sep 06 14:10:37.984724 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.984684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e093cad-6f89-4c15-8391-928c3dc25d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf\" (UID: \"5e093cad-6f89-4c15-8391-928c3dc25d79\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" Sep 06 14:10:37.984900 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.984784 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b149c687-9e38-46a3-90d7-b9b6201b2700-plugins-config-volume\") pod \"payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx\" (UID: \"b149c687-9e38-46a3-90d7-b9b6201b2700\") " pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" Sep 06 14:10:37.984900 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.984820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtf8\" (UniqueName: \"kubernetes.io/projected/5e093cad-6f89-4c15-8391-928c3dc25d79-kube-api-access-9mtf8\") pod \"payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf\" (UID: \"5e093cad-6f89-4c15-8391-928c3dc25d79\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" Sep 06 14:10:37.985050 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.984939 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsgtl\" (UniqueName: \"kubernetes.io/projected/b149c687-9e38-46a3-90d7-b9b6201b2700-kube-api-access-lsgtl\") pod \"payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx\" (UID: \"b149c687-9e38-46a3-90d7-b9b6201b2700\") " pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" Sep 06 14:10:37.985557 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.985533 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e093cad-6f89-4c15-8391-928c3dc25d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf\" (UID: \"5e093cad-6f89-4c15-8391-928c3dc25d79\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" Sep 06 14:10:37.985733 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.985711 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b149c687-9e38-46a3-90d7-b9b6201b2700-plugins-config-volume\") pod \"payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx\" (UID: \"b149c687-9e38-46a3-90d7-b9b6201b2700\") " pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" Sep 06 14:10:37.993151 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.993124 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtf8\" (UniqueName: \"kubernetes.io/projected/5e093cad-6f89-4c15-8391-928c3dc25d79-kube-api-access-9mtf8\") pod \"payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf\" (UID: \"5e093cad-6f89-4c15-8391-928c3dc25d79\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" Sep 06 14:10:37.993290 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:37.993268 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsgtl\" (UniqueName: \"kubernetes.io/projected/b149c687-9e38-46a3-90d7-b9b6201b2700-kube-api-access-lsgtl\") pod \"payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx\" (UID: \"b149c687-9e38-46a3-90d7-b9b6201b2700\") " pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" Sep 06 14:10:38.013018 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:38.012999 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" Sep 06 14:10:38.053237 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:38.053038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" Sep 06 14:10:38.291272 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:38.291250 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf"] Sep 06 14:10:38.292992 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:10:38.292950 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e093cad_6f89_4c15_8391_928c3dc25d79.slice/crio-6892f76b2509b080a967e0029ea83e8b391b2f4f2a4f07fc69f7ff4ea6e591e8 WatchSource:0}: Error finding container 6892f76b2509b080a967e0029ea83e8b391b2f4f2a4f07fc69f7ff4ea6e591e8: Status 404 returned error can't find the container with id 6892f76b2509b080a967e0029ea83e8b391b2f4f2a4f07fc69f7ff4ea6e591e8 Sep 06 14:10:38.337037 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:38.337011 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx"] Sep 06 14:10:38.349946 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:10:38.349924 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb149c687_9e38_46a3_90d7_b9b6201b2700.slice/crio-c9764d08572ec69da387a22270d6eea65fa822fa575ffb5f5f6a1050b587ef1e WatchSource:0}: Error finding container c9764d08572ec69da387a22270d6eea65fa822fa575ffb5f5f6a1050b587ef1e: Status 404 returned error can't find the container with id c9764d08572ec69da387a22270d6eea65fa822fa575ffb5f5f6a1050b587ef1e Sep 06 14:10:38.537693 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:38.537651 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:38.636348 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:38.636257 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:38.978667 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:38.978558 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" event={"ID":"5e093cad-6f89-4c15-8391-928c3dc25d79","Type":"ContainerStarted","Data":"3ccd1c33bb6b9ac06b4aac908cd9901aa17d9d88ae9aae8bd5b387d3c6112e4b"} Sep 06 14:10:38.978667 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:38.978642 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" event={"ID":"5e093cad-6f89-4c15-8391-928c3dc25d79","Type":"ContainerStarted","Data":"6892f76b2509b080a967e0029ea83e8b391b2f4f2a4f07fc69f7ff4ea6e591e8"} Sep 06 14:10:38.978943 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:38.978672 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" Sep 06 14:10:38.980350 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:38.980315 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" event={"ID":"b149c687-9e38-46a3-90d7-b9b6201b2700","Type":"ContainerStarted","Data":"73f1961e84f33d52cea789526bc2dfa873239f555400f78defe3ca7fb05beeaa"} Sep 06 14:10:38.980502 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:38.980371 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" event={"ID":"b149c687-9e38-46a3-90d7-b9b6201b2700","Type":"ContainerStarted","Data":"c9764d08572ec69da387a22270d6eea65fa822fa575ffb5f5f6a1050b587ef1e"} Sep 06 14:10:38.980502 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:38.980418 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" Sep 06 14:10:38.997461 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:38.997414 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" podStartSLOduration=1.997400722 podStartE2EDuration="1.997400722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:38.996062154 +0000 UTC m=+770.451551015" watchObservedRunningTime="2026-09-06 14:10:38.997400722 +0000 UTC m=+770.452889584" Sep 06 14:10:39.016868 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:39.016823 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" podStartSLOduration=2.016810261 podStartE2EDuration="2.016810261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:39.015669271 +0000 UTC m=+770.471158134" watchObservedRunningTime="2026-09-06 14:10:39.016810261 +0000 UTC m=+770.472299114" Sep 06 14:10:39.725279 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:39.725246 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:39.825419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:39.825359 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:39.949073 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:39.949031 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:40.160401 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.160364 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:40.264221 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.264193 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:40.274734 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.274708 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64"] Sep 06 14:10:40.278368 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.278354 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" Sep 06 14:10:40.280646 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.280626 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-1482ae\"" Sep 06 14:10:40.280753 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.280677 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-1482ae-dockercfg-2pn6c\"" Sep 06 14:10:40.287359 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.287314 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64"] Sep 06 14:10:40.304010 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.303987 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn"] Sep 06 14:10:40.307471 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.307454 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" Sep 06 14:10:40.317212 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.317190 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn"] Sep 06 14:10:40.411454 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.411375 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64788558-c059-4cda-a753-64d3393bfd88-plugins-config-volume\") pod \"payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn\" (UID: \"64788558-c059-4cda-a753-64d3393bfd88\") " pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" Sep 06 14:10:40.411454 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.411422 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl45z\" (UniqueName: \"kubernetes.io/projected/64788558-c059-4cda-a753-64d3393bfd88-kube-api-access-rl45z\") pod \"payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn\" (UID: \"64788558-c059-4cda-a753-64d3393bfd88\") " pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" Sep 06 14:10:40.411651 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.411528 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75s2f\" (UniqueName: \"kubernetes.io/projected/017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9-kube-api-access-75s2f\") pod \"payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64\" (UID: \"017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" Sep 06 14:10:40.411651 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.411568 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64\" (UID: \"017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" Sep 06 14:10:40.512147 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.512112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64\" (UID: \"017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" Sep 06 14:10:40.512306 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.512217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64788558-c059-4cda-a753-64d3393bfd88-plugins-config-volume\") pod \"payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn\" (UID: \"64788558-c059-4cda-a753-64d3393bfd88\") " pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" Sep 06 14:10:40.512306 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.512249 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl45z\" (UniqueName: \"kubernetes.io/projected/64788558-c059-4cda-a753-64d3393bfd88-kube-api-access-rl45z\") pod \"payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn\" (UID: \"64788558-c059-4cda-a753-64d3393bfd88\") " pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" Sep 06 14:10:40.512384 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.512301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75s2f\" (UniqueName: \"kubernetes.io/projected/017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9-kube-api-access-75s2f\") pod \"payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64\" (UID: \"017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" Sep 06 14:10:40.512809 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.512788 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64\" (UID: \"017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" Sep 06 14:10:40.513012 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.512996 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64788558-c059-4cda-a753-64d3393bfd88-plugins-config-volume\") pod \"payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn\" (UID: \"64788558-c059-4cda-a753-64d3393bfd88\") " pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" Sep 06 14:10:40.520086 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.520065 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75s2f\" (UniqueName: \"kubernetes.io/projected/017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9-kube-api-access-75s2f\") pod \"payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64\" (UID: \"017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" Sep 06 14:10:40.520220 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.520192 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl45z\" (UniqueName: \"kubernetes.io/projected/64788558-c059-4cda-a753-64d3393bfd88-kube-api-access-rl45z\") pod \"payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn\" (UID: \"64788558-c059-4cda-a753-64d3393bfd88\") " pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" Sep 06 14:10:40.591586 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.591547 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" Sep 06 14:10:40.618282 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.618255 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" Sep 06 14:10:40.870928 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.870889 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64"] Sep 06 14:10:40.873328 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:10:40.873304 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017b9ae0_0d9e_42cf_88f3_28aa6ab81ab9.slice/crio-21b9d6fdaf18d6b5b8fc2fadc68b8475975f88b746721bc8804e99ec90340bc3 WatchSource:0}: Error finding container 21b9d6fdaf18d6b5b8fc2fadc68b8475975f88b746721bc8804e99ec90340bc3: Status 404 returned error can't find the container with id 21b9d6fdaf18d6b5b8fc2fadc68b8475975f88b746721bc8804e99ec90340bc3 Sep 06 14:10:40.896555 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.896534 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn"] Sep 06 14:10:40.912096 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:10:40.912071 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64788558_c059_4cda_a753_64d3393bfd88.slice/crio-e6b56798002fa117423da05068fe3b9b3bcf2289a416770b27f30b448bae2ccf WatchSource:0}: Error finding container e6b56798002fa117423da05068fe3b9b3bcf2289a416770b27f30b448bae2ccf: Status 404 returned error can't find the container with id e6b56798002fa117423da05068fe3b9b3bcf2289a416770b27f30b448bae2ccf Sep 06 14:10:40.994181 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.994143 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" event={"ID":"017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9","Type":"ContainerStarted","Data":"94a64491cb6311ec0d964202743f7638ca4682939974f567ea0ba6c6da0dafef"} Sep 06 14:10:40.994328 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.994209 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" event={"ID":"017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9","Type":"ContainerStarted","Data":"21b9d6fdaf18d6b5b8fc2fadc68b8475975f88b746721bc8804e99ec90340bc3"} Sep 06 14:10:40.994328 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.994242 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" Sep 06 14:10:40.995719 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.995694 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" event={"ID":"64788558-c059-4cda-a753-64d3393bfd88","Type":"ContainerStarted","Data":"93232882a31dad62c1bbbceec4997e3d7d43c2c54e937abe0247e4407c47dc2d"} Sep 06 14:10:40.995812 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.995741 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" event={"ID":"64788558-c059-4cda-a753-64d3393bfd88","Type":"ContainerStarted","Data":"e6b56798002fa117423da05068fe3b9b3bcf2289a416770b27f30b448bae2ccf"} Sep 06 14:10:40.995812 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:40.995805 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" Sep 06 14:10:41.010028 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:41.009977 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" podStartSLOduration=1.009957031 podStartE2EDuration="1.009957031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:41.008870296 +0000 UTC m=+772.464359160" watchObservedRunningTime="2026-09-06 14:10:41.009957031 +0000 UTC m=+772.465445888" Sep 06 14:10:41.024832 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:41.024788 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" podStartSLOduration=1.024764181 podStartE2EDuration="1.024764181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:41.022698102 +0000 UTC m=+772.478186965" watchObservedRunningTime="2026-09-06 14:10:41.024764181 +0000 UTC m=+772.480253043" Sep 06 14:10:41.928824 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:41.928790 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" Sep 06 14:10:41.929274 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:41.928864 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" Sep 06 14:10:42.204358 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:42.204281 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:10:45.860115 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:45.860073 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:46.002873 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:46.002813 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:46.096507 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:46.096466 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:46.199344 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:46.199241 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:46.298184 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:46.298149 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:46.456323 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:46.456241 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:48.834412 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:48.834380 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:48.987313 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:48.987274 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:49.987086 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:49.987055 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" Sep 06 14:10:49.987549 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:49.987214 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" Sep 06 14:10:50.420263 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:50.420230 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:51.616251 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:51.616222 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:52.004410 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:52.004333 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" Sep 06 14:10:52.004410 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:52.004398 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" Sep 06 14:10:52.477486 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:52.477452 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:52.537749 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:52.537720 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:52.619237 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:52.619202 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:52.800207 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:52.800178 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:53.420228 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:53.420199 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:53.723315 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:53.723235 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:54.003397 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:54.003313 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:54.300289 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:54.300251 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:56.563551 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:56.563512 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:56.662184 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:56.662152 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:57.718376 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:57.718338 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:57.820185 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:57.820149 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:58.152940 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.152895 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:58.260467 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.260437 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:58.347692 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.347663 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:58.352830 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.352806 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s"] Sep 06 14:10:58.356734 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.356716 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" Sep 06 14:10:58.359158 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.359138 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5ec585-dockercfg-k5d5v\"" Sep 06 14:10:58.359257 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.359216 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5ec585\"" Sep 06 14:10:58.368012 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.367991 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s"] Sep 06 14:10:58.475963 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.475865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e110340-3eed-4d79-8191-bc66e1549897-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s\" (UID: \"1e110340-3eed-4d79-8191-bc66e1549897\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" Sep 06 14:10:58.475963 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.475926 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh8zj\" (UniqueName: \"kubernetes.io/projected/1e110340-3eed-4d79-8191-bc66e1549897-kube-api-access-fh8zj\") pod \"payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s\" (UID: \"1e110340-3eed-4d79-8191-bc66e1549897\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" Sep 06 14:10:58.577234 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.577203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e110340-3eed-4d79-8191-bc66e1549897-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s\" (UID: \"1e110340-3eed-4d79-8191-bc66e1549897\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" Sep 06 14:10:58.577234 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.577235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh8zj\" (UniqueName: \"kubernetes.io/projected/1e110340-3eed-4d79-8191-bc66e1549897-kube-api-access-fh8zj\") pod \"payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s\" (UID: \"1e110340-3eed-4d79-8191-bc66e1549897\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" Sep 06 14:10:58.578171 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.578150 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e110340-3eed-4d79-8191-bc66e1549897-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s\" (UID: \"1e110340-3eed-4d79-8191-bc66e1549897\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" Sep 06 14:10:58.584458 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.584427 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh8zj\" (UniqueName: \"kubernetes.io/projected/1e110340-3eed-4d79-8191-bc66e1549897-kube-api-access-fh8zj\") pod \"payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s\" (UID: \"1e110340-3eed-4d79-8191-bc66e1549897\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" Sep 06 14:10:58.668591 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.668557 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" Sep 06 14:10:58.725156 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.725126 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:58.936484 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:58.936462 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s"] Sep 06 14:10:58.938635 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:10:58.938604 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e110340_3eed_4d79_8191_bc66e1549897.slice/crio-05822eb0a95210be56591a6865d8b4b317fe89a911656d7b89145a095e3937a5 WatchSource:0}: Error finding container 05822eb0a95210be56591a6865d8b4b317fe89a911656d7b89145a095e3937a5: Status 404 returned error can't find the container with id 05822eb0a95210be56591a6865d8b4b317fe89a911656d7b89145a095e3937a5 Sep 06 14:10:59.032349 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:59.032320 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:59.094684 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:59.094656 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" event={"ID":"1e110340-3eed-4d79-8191-bc66e1549897","Type":"ContainerStarted","Data":"206bb4b9eeaf9f3024121b29bf68774a75c3eb3c695c8fad45d3e02c22548697"} Sep 06 14:10:59.094824 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:59.094701 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" event={"ID":"1e110340-3eed-4d79-8191-bc66e1549897","Type":"ContainerStarted","Data":"05822eb0a95210be56591a6865d8b4b317fe89a911656d7b89145a095e3937a5"} Sep 06 14:10:59.094824 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:59.094743 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" Sep 06 14:10:59.110040 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:59.109988 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" podStartSLOduration=1.109973117 podStartE2EDuration="1.109973117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:59.109462481 +0000 UTC m=+790.564951343" watchObservedRunningTime="2026-09-06 14:10:59.109973117 +0000 UTC m=+790.565462005" Sep 06 14:10:59.116282 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:59.116258 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:10:59.202086 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:59.202015 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:10:59.540674 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:10:59.540619 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:00.410646 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:00.410617 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:01.310645 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:01.310617 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:01.930037 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:01.930002 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:02.603053 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:02.603019 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:03.402575 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:03.402536 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:10.102672 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:10.102637 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" Sep 06 14:11:15.485429 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:15.485353 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:11:17.305370 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:17.305339 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:17.842120 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:17.842089 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:17.867129 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:17.867094 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx"] Sep 06 14:11:17.867377 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:17.867350 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" podUID="b149c687-9e38-46a3-90d7-b9b6201b2700" containerName="payload-processing" containerID="cri-o://73f1961e84f33d52cea789526bc2dfa873239f555400f78defe3ca7fb05beeaa" gracePeriod=30 Sep 06 14:11:17.875652 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:17.875629 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf"] Sep 06 14:11:17.876239 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:17.876015 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" podUID="5e093cad-6f89-4c15-8391-928c3dc25d79" containerName="payload-pre-processing" containerID="cri-o://3ccd1c33bb6b9ac06b4aac908cd9901aa17d9d88ae9aae8bd5b387d3c6112e4b" gracePeriod=30 Sep 06 14:11:17.907565 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:17.907540 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:17.963780 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:17.963742 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:18.021167 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.021127 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:18.158980 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.158945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:18.187597 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.187571 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" Sep 06 14:11:18.191034 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.191014 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" Sep 06 14:11:18.191874 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.191851 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e093cad-6f89-4c15-8391-928c3dc25d79" containerID="3ccd1c33bb6b9ac06b4aac908cd9901aa17d9d88ae9aae8bd5b387d3c6112e4b" exitCode=0 Sep 06 14:11:18.191976 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.191931 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" event={"ID":"5e093cad-6f89-4c15-8391-928c3dc25d79","Type":"ContainerDied","Data":"3ccd1c33bb6b9ac06b4aac908cd9901aa17d9d88ae9aae8bd5b387d3c6112e4b"} Sep 06 14:11:18.192023 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.192000 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" event={"ID":"5e093cad-6f89-4c15-8391-928c3dc25d79","Type":"ContainerDied","Data":"6892f76b2509b080a967e0029ea83e8b391b2f4f2a4f07fc69f7ff4ea6e591e8"} Sep 06 14:11:18.192064 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.192030 2577 scope.go:117] "RemoveContainer" containerID="3ccd1c33bb6b9ac06b4aac908cd9901aa17d9d88ae9aae8bd5b387d3c6112e4b" Sep 06 14:11:18.193149 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.193130 2577 generic.go:356] "Generic (PLEG): container finished" podID="b149c687-9e38-46a3-90d7-b9b6201b2700" containerID="73f1961e84f33d52cea789526bc2dfa873239f555400f78defe3ca7fb05beeaa" exitCode=0 Sep 06 14:11:18.193219 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.193169 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" Sep 06 14:11:18.193258 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.193216 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" event={"ID":"b149c687-9e38-46a3-90d7-b9b6201b2700","Type":"ContainerDied","Data":"73f1961e84f33d52cea789526bc2dfa873239f555400f78defe3ca7fb05beeaa"} Sep 06 14:11:18.193304 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.193254 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx" event={"ID":"b149c687-9e38-46a3-90d7-b9b6201b2700","Type":"ContainerDied","Data":"c9764d08572ec69da387a22270d6eea65fa822fa575ffb5f5f6a1050b587ef1e"} Sep 06 14:11:18.200305 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.200288 2577 scope.go:117] "RemoveContainer" containerID="3ccd1c33bb6b9ac06b4aac908cd9901aa17d9d88ae9aae8bd5b387d3c6112e4b" Sep 06 14:11:18.200564 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:11:18.200546 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ccd1c33bb6b9ac06b4aac908cd9901aa17d9d88ae9aae8bd5b387d3c6112e4b\": container with ID starting with 3ccd1c33bb6b9ac06b4aac908cd9901aa17d9d88ae9aae8bd5b387d3c6112e4b not found: ID does not exist" containerID="3ccd1c33bb6b9ac06b4aac908cd9901aa17d9d88ae9aae8bd5b387d3c6112e4b" Sep 06 14:11:18.200605 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.200571 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ccd1c33bb6b9ac06b4aac908cd9901aa17d9d88ae9aae8bd5b387d3c6112e4b"} err="failed to get container status \"3ccd1c33bb6b9ac06b4aac908cd9901aa17d9d88ae9aae8bd5b387d3c6112e4b\": rpc error: code = NotFound desc = could not find container \"3ccd1c33bb6b9ac06b4aac908cd9901aa17d9d88ae9aae8bd5b387d3c6112e4b\": container with ID starting with 3ccd1c33bb6b9ac06b4aac908cd9901aa17d9d88ae9aae8bd5b387d3c6112e4b not found: ID does not exist" Sep 06 14:11:18.200605 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.200585 2577 scope.go:117] "RemoveContainer" containerID="73f1961e84f33d52cea789526bc2dfa873239f555400f78defe3ca7fb05beeaa" Sep 06 14:11:18.210106 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.210087 2577 scope.go:117] "RemoveContainer" containerID="73f1961e84f33d52cea789526bc2dfa873239f555400f78defe3ca7fb05beeaa" Sep 06 14:11:18.210427 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:11:18.210405 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f1961e84f33d52cea789526bc2dfa873239f555400f78defe3ca7fb05beeaa\": container with ID starting with 73f1961e84f33d52cea789526bc2dfa873239f555400f78defe3ca7fb05beeaa not found: ID does not exist" containerID="73f1961e84f33d52cea789526bc2dfa873239f555400f78defe3ca7fb05beeaa" Sep 06 14:11:18.210478 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.210439 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f1961e84f33d52cea789526bc2dfa873239f555400f78defe3ca7fb05beeaa"} err="failed to get container status \"73f1961e84f33d52cea789526bc2dfa873239f555400f78defe3ca7fb05beeaa\": rpc error: code = NotFound desc = could not find container \"73f1961e84f33d52cea789526bc2dfa873239f555400f78defe3ca7fb05beeaa\": container with ID starting with 73f1961e84f33d52cea789526bc2dfa873239f555400f78defe3ca7fb05beeaa not found: ID does not exist" Sep 06 14:11:18.267398 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.267364 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsgtl\" (UniqueName: \"kubernetes.io/projected/b149c687-9e38-46a3-90d7-b9b6201b2700-kube-api-access-lsgtl\") pod \"b149c687-9e38-46a3-90d7-b9b6201b2700\" (UID: \"b149c687-9e38-46a3-90d7-b9b6201b2700\") " Sep 06 14:11:18.267586 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.267459 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b149c687-9e38-46a3-90d7-b9b6201b2700-plugins-config-volume\") pod \"b149c687-9e38-46a3-90d7-b9b6201b2700\" (UID: \"b149c687-9e38-46a3-90d7-b9b6201b2700\") " Sep 06 14:11:18.267769 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.267749 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b149c687-9e38-46a3-90d7-b9b6201b2700-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b149c687-9e38-46a3-90d7-b9b6201b2700" (UID: "b149c687-9e38-46a3-90d7-b9b6201b2700"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:11:18.267891 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.267878 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b149c687-9e38-46a3-90d7-b9b6201b2700-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:18.269663 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.269636 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b149c687-9e38-46a3-90d7-b9b6201b2700-kube-api-access-lsgtl" (OuterVolumeSpecName: "kube-api-access-lsgtl") pod "b149c687-9e38-46a3-90d7-b9b6201b2700" (UID: "b149c687-9e38-46a3-90d7-b9b6201b2700"). InnerVolumeSpecName "kube-api-access-lsgtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:11:18.369156 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.369123 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mtf8\" (UniqueName: \"kubernetes.io/projected/5e093cad-6f89-4c15-8391-928c3dc25d79-kube-api-access-9mtf8\") pod \"5e093cad-6f89-4c15-8391-928c3dc25d79\" (UID: \"5e093cad-6f89-4c15-8391-928c3dc25d79\") " Sep 06 14:11:18.369623 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.369184 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e093cad-6f89-4c15-8391-928c3dc25d79-plugins-config-volume\") pod \"5e093cad-6f89-4c15-8391-928c3dc25d79\" (UID: \"5e093cad-6f89-4c15-8391-928c3dc25d79\") " Sep 06 14:11:18.369623 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.369521 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsgtl\" (UniqueName: \"kubernetes.io/projected/b149c687-9e38-46a3-90d7-b9b6201b2700-kube-api-access-lsgtl\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:18.369722 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.369626 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e093cad-6f89-4c15-8391-928c3dc25d79-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e093cad-6f89-4c15-8391-928c3dc25d79" (UID: "5e093cad-6f89-4c15-8391-928c3dc25d79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:11:18.371281 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.371250 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e093cad-6f89-4c15-8391-928c3dc25d79-kube-api-access-9mtf8" (OuterVolumeSpecName: "kube-api-access-9mtf8") pod "5e093cad-6f89-4c15-8391-928c3dc25d79" (UID: "5e093cad-6f89-4c15-8391-928c3dc25d79"). InnerVolumeSpecName "kube-api-access-9mtf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:11:18.470931 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.470846 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mtf8\" (UniqueName: \"kubernetes.io/projected/5e093cad-6f89-4c15-8391-928c3dc25d79-kube-api-access-9mtf8\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:18.470931 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.470871 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e093cad-6f89-4c15-8391-928c3dc25d79-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:18.514113 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.514083 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx"] Sep 06 14:11:18.519286 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.519263 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84753aa3-79464f9f8c-pmjjx"] Sep 06 14:11:18.587695 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.587617 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:18.685569 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.685538 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:18.788077 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.788028 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:18.850535 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:18.850510 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:19.069482 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.069401 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b149c687-9e38-46a3-90d7-b9b6201b2700" path="/var/lib/kubelet/pods/b149c687-9e38-46a3-90d7-b9b6201b2700/volumes" Sep 06 14:11:19.149216 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.149186 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv"] Sep 06 14:11:19.149659 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.149644 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e093cad-6f89-4c15-8391-928c3dc25d79" containerName="payload-pre-processing" Sep 06 14:11:19.149710 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.149662 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e093cad-6f89-4c15-8391-928c3dc25d79" containerName="payload-pre-processing" Sep 06 14:11:19.149710 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.149675 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b149c687-9e38-46a3-90d7-b9b6201b2700" containerName="payload-processing" Sep 06 14:11:19.149710 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.149682 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b149c687-9e38-46a3-90d7-b9b6201b2700" containerName="payload-processing" Sep 06 14:11:19.149812 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.149753 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e093cad-6f89-4c15-8391-928c3dc25d79" containerName="payload-pre-processing" Sep 06 14:11:19.149812 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.149761 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b149c687-9e38-46a3-90d7-b9b6201b2700" containerName="payload-processing" Sep 06 14:11:19.154499 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.154475 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.157292 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.157272 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c0e08d9a-gw-openshift-default-dockercfg-5zxxt\"" Sep 06 14:11:19.169391 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.169373 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv"] Sep 06 14:11:19.184363 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.184342 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:19.199858 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.199835 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf" Sep 06 14:11:19.215254 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.215223 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf"] Sep 06 14:11:19.219173 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.219138 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84753aa3-5f9f69bf58-w65qf"] Sep 06 14:11:19.277772 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.277734 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-istio-envoy\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.277974 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.277855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wxjs\" (UniqueName: \"kubernetes.io/projected/d38569a3-7eda-465b-be47-ddafa712a99d-kube-api-access-2wxjs\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.277974 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.277899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-istio-data\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.277974 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.277964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-workload-certs\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.278152 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.278016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d38569a3-7eda-465b-be47-ddafa712a99d-istio-token\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.278152 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.278054 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-workload-socket\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.278152 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.278102 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d38569a3-7eda-465b-be47-ddafa712a99d-istiod-ca-cert\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.278241 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.278167 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d38569a3-7eda-465b-be47-ddafa712a99d-istio-podinfo\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.278241 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.278214 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-credential-socket\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.295824 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.295793 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:19.379420 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.379328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-istio-envoy\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.379877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.379416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wxjs\" (UniqueName: \"kubernetes.io/projected/d38569a3-7eda-465b-be47-ddafa712a99d-kube-api-access-2wxjs\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.379877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.379451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-istio-data\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.379877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.379488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-workload-certs\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.379877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.379531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d38569a3-7eda-465b-be47-ddafa712a99d-istio-token\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.379877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.379568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-workload-socket\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.379877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.379617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d38569a3-7eda-465b-be47-ddafa712a99d-istiod-ca-cert\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.379877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.379658 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d38569a3-7eda-465b-be47-ddafa712a99d-istio-podinfo\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.379877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.379696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-credential-socket\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.379877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.379847 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-istio-data\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.380285 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.379901 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-workload-certs\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.380285 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.380173 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-workload-socket\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.380461 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.380440 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-credential-socket\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.380517 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.380468 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d38569a3-7eda-465b-be47-ddafa712a99d-istiod-ca-cert\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.382210 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.382185 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-istio-envoy\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.382362 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.382344 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d38569a3-7eda-465b-be47-ddafa712a99d-istio-podinfo\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.386824 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.386795 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d38569a3-7eda-465b-be47-ddafa712a99d-istio-token\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.387050 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.387032 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wxjs\" (UniqueName: \"kubernetes.io/projected/d38569a3-7eda-465b-be47-ddafa712a99d-kube-api-access-2wxjs\") pod \"e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.465778 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.465746 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:19.749088 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.749012 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv"] Sep 06 14:11:19.753011 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:11:19.752964 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38569a3_7eda_465b_be47_ddafa712a99d.slice/crio-4222ef1bd49d4379683a75416d7973a96fe490ef0be3708accfd108537d8c2f1 WatchSource:0}: Error finding container 4222ef1bd49d4379683a75416d7973a96fe490ef0be3708accfd108537d8c2f1: Status 404 returned error can't find the container with id 4222ef1bd49d4379683a75416d7973a96fe490ef0be3708accfd108537d8c2f1 Sep 06 14:11:19.755481 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.755438 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:11:19.755588 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.755531 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:11:19.755588 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:19.755577 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:11:20.207896 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:20.207857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" event={"ID":"d38569a3-7eda-465b-be47-ddafa712a99d","Type":"ContainerStarted","Data":"3d361057493f3bf2143dbe87d4b944d1b3415575f451752a41fc1a657cdd6e8e"} Sep 06 14:11:20.208112 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:20.207929 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" event={"ID":"d38569a3-7eda-465b-be47-ddafa712a99d","Type":"ContainerStarted","Data":"4222ef1bd49d4379683a75416d7973a96fe490ef0be3708accfd108537d8c2f1"} Sep 06 14:11:20.226262 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:20.226214 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" podStartSLOduration=1.226199793 podStartE2EDuration="1.226199793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:11:20.225667578 +0000 UTC m=+811.681156440" watchObservedRunningTime="2026-09-06 14:11:20.226199793 +0000 UTC m=+811.681688715" Sep 06 14:11:20.466114 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:20.466039 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:20.471182 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:20.471156 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:21.068361 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:21.068330 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e093cad-6f89-4c15-8391-928c3dc25d79" path="/var/lib/kubelet/pods/5e093cad-6f89-4c15-8391-928c3dc25d79/volumes" Sep 06 14:11:21.212470 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:21.212435 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:21.213330 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:21.213312 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:21.469515 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:21.469441 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:21.661345 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:21.661295 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:21.760323 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:21.760243 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:21.860741 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:21.860708 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:21.960161 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:21.960131 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:22.175421 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.175386 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:22.281260 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.281229 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:22.377253 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.377224 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:22.378385 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.378362 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx"] Sep 06 14:11:22.382315 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.382298 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" Sep 06 14:11:22.384500 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.384481 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c0e08d9a-dockercfg-llmpm\"" Sep 06 14:11:22.384621 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.384496 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c0e08d9a\"" Sep 06 14:11:22.393086 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.393064 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx"] Sep 06 14:11:22.407167 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.407143 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c"] Sep 06 14:11:22.410811 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.410794 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" Sep 06 14:11:22.421410 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.421392 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c"] Sep 06 14:11:22.515473 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.515392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz8rm\" (UniqueName: \"kubernetes.io/projected/995e8174-b715-4dd4-b913-e363072f6529-kube-api-access-rz8rm\") pod \"payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c\" (UID: \"995e8174-b715-4dd4-b913-e363072f6529\") " pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" Sep 06 14:11:22.515473 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.515428 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llfm4\" (UniqueName: \"kubernetes.io/projected/0588afea-89c3-44cd-bbcb-05dc9a1ce41b-kube-api-access-llfm4\") pod \"payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx\" (UID: \"0588afea-89c3-44cd-bbcb-05dc9a1ce41b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" Sep 06 14:11:22.515857 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.515563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0588afea-89c3-44cd-bbcb-05dc9a1ce41b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx\" (UID: \"0588afea-89c3-44cd-bbcb-05dc9a1ce41b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" Sep 06 14:11:22.515857 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.515605 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/995e8174-b715-4dd4-b913-e363072f6529-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c\" (UID: \"995e8174-b715-4dd4-b913-e363072f6529\") " pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" Sep 06 14:11:22.616935 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.616881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0588afea-89c3-44cd-bbcb-05dc9a1ce41b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx\" (UID: \"0588afea-89c3-44cd-bbcb-05dc9a1ce41b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" Sep 06 14:11:22.617090 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.616983 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/995e8174-b715-4dd4-b913-e363072f6529-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c\" (UID: \"995e8174-b715-4dd4-b913-e363072f6529\") " pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" Sep 06 14:11:22.617090 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.617041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz8rm\" (UniqueName: \"kubernetes.io/projected/995e8174-b715-4dd4-b913-e363072f6529-kube-api-access-rz8rm\") pod \"payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c\" (UID: \"995e8174-b715-4dd4-b913-e363072f6529\") " pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" Sep 06 14:11:22.617090 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.617065 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfm4\" (UniqueName: \"kubernetes.io/projected/0588afea-89c3-44cd-bbcb-05dc9a1ce41b-kube-api-access-llfm4\") pod \"payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx\" (UID: \"0588afea-89c3-44cd-bbcb-05dc9a1ce41b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" Sep 06 14:11:22.617563 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.617544 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0588afea-89c3-44cd-bbcb-05dc9a1ce41b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx\" (UID: \"0588afea-89c3-44cd-bbcb-05dc9a1ce41b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" Sep 06 14:11:22.617719 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.617701 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/995e8174-b715-4dd4-b913-e363072f6529-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c\" (UID: \"995e8174-b715-4dd4-b913-e363072f6529\") " pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" Sep 06 14:11:22.624303 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.624283 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz8rm\" (UniqueName: \"kubernetes.io/projected/995e8174-b715-4dd4-b913-e363072f6529-kube-api-access-rz8rm\") pod \"payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c\" (UID: \"995e8174-b715-4dd4-b913-e363072f6529\") " pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" Sep 06 14:11:22.624482 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.624462 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llfm4\" (UniqueName: \"kubernetes.io/projected/0588afea-89c3-44cd-bbcb-05dc9a1ce41b-kube-api-access-llfm4\") pod \"payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx\" (UID: \"0588afea-89c3-44cd-bbcb-05dc9a1ce41b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" Sep 06 14:11:22.693711 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.693686 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" Sep 06 14:11:22.722024 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.722001 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" Sep 06 14:11:22.974106 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:22.974038 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx"] Sep 06 14:11:22.978659 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:11:22.978632 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0588afea_89c3_44cd_bbcb_05dc9a1ce41b.slice/crio-7b9c1c11b12d5336dfebed4b6febe27ffd6434540667a606cd30c86c218843ff WatchSource:0}: Error finding container 7b9c1c11b12d5336dfebed4b6febe27ffd6434540667a606cd30c86c218843ff: Status 404 returned error can't find the container with id 7b9c1c11b12d5336dfebed4b6febe27ffd6434540667a606cd30c86c218843ff Sep 06 14:11:23.006293 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:23.006271 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c"] Sep 06 14:11:23.008357 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:11:23.008335 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995e8174_b715_4dd4_b913_e363072f6529.slice/crio-a7f92c9c90a2c514ba2d6c2a0fed700d8d365e08ca1f89d88dadb071ad77b0af WatchSource:0}: Error finding container a7f92c9c90a2c514ba2d6c2a0fed700d8d365e08ca1f89d88dadb071ad77b0af: Status 404 returned error can't find the container with id a7f92c9c90a2c514ba2d6c2a0fed700d8d365e08ca1f89d88dadb071ad77b0af Sep 06 14:11:23.223951 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:23.223891 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" event={"ID":"0588afea-89c3-44cd-bbcb-05dc9a1ce41b","Type":"ContainerStarted","Data":"522ac21a1076af6c2a21873a1fe5a1d23e855d6bc7672e327ecac89ca8f360f1"} Sep 06 14:11:23.224125 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:23.223972 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" event={"ID":"0588afea-89c3-44cd-bbcb-05dc9a1ce41b","Type":"ContainerStarted","Data":"7b9c1c11b12d5336dfebed4b6febe27ffd6434540667a606cd30c86c218843ff"} Sep 06 14:11:23.224125 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:23.224003 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" Sep 06 14:11:23.225742 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:23.225709 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" event={"ID":"995e8174-b715-4dd4-b913-e363072f6529","Type":"ContainerStarted","Data":"2440369e2c194b4b002e7e30285226ea612024be66414640d59d99af881e7e41"} Sep 06 14:11:23.225876 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:23.225762 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" event={"ID":"995e8174-b715-4dd4-b913-e363072f6529","Type":"ContainerStarted","Data":"a7f92c9c90a2c514ba2d6c2a0fed700d8d365e08ca1f89d88dadb071ad77b0af"} Sep 06 14:11:23.225876 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:23.225807 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" Sep 06 14:11:23.241816 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:23.241770 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" podStartSLOduration=1.241756459 podStartE2EDuration="1.241756459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:11:23.2394953 +0000 UTC m=+814.694984163" watchObservedRunningTime="2026-09-06 14:11:23.241756459 +0000 UTC m=+814.697245348" Sep 06 14:11:23.256246 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:23.256200 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" podStartSLOduration=1.256182479 podStartE2EDuration="1.256182479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:11:23.254593819 +0000 UTC m=+814.710082683" watchObservedRunningTime="2026-09-06 14:11:23.256182479 +0000 UTC m=+814.711671346" Sep 06 14:11:34.232785 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:34.232748 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" Sep 06 14:11:34.233238 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:34.232822 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" Sep 06 14:11:34.499967 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:34.499878 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:11:42.814749 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:42.814713 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:45.345641 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:45.345603 2577 generic.go:356] "Generic (PLEG): container finished" podID="d38569a3-7eda-465b-be47-ddafa712a99d" containerID="3d361057493f3bf2143dbe87d4b944d1b3415575f451752a41fc1a657cdd6e8e" exitCode=0 Sep 06 14:11:45.346067 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:45.345671 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" event={"ID":"d38569a3-7eda-465b-be47-ddafa712a99d","Type":"ContainerDied","Data":"3d361057493f3bf2143dbe87d4b944d1b3415575f451752a41fc1a657cdd6e8e"} Sep 06 14:11:45.346067 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:45.346047 2577 scope.go:117] "RemoveContainer" containerID="3d361057493f3bf2143dbe87d4b944d1b3415575f451752a41fc1a657cdd6e8e" Sep 06 14:11:45.347178 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:45.347147 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:11:45.347277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:45.347231 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:11:45.347313 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:45.347277 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:11:46.356633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:46.356599 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" event={"ID":"d38569a3-7eda-465b-be47-ddafa712a99d","Type":"ContainerStarted","Data":"895d36df299067fefbd8a15b44dc5acaaf2eb7ebae88b9b402ad735df3a6a6c4"} Sep 06 14:11:46.466780 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:46.466747 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:46.467873 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:46.467843 2577 patch_prober.go:28] interesting pod/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 06 14:11:46.468013 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:46.467894 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" podUID="d38569a3-7eda-465b-be47-ddafa712a99d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 06 14:11:47.467181 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:47.467138 2577 patch_prober.go:28] interesting pod/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 06 14:11:47.467648 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:47.467204 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" podUID="d38569a3-7eda-465b-be47-ddafa712a99d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 06 14:11:48.466307 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:48.466264 2577 patch_prober.go:28] interesting pod/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 06 14:11:48.466517 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:48.466326 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" podUID="d38569a3-7eda-465b-be47-ddafa712a99d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 06 14:11:49.466519 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:49.466425 2577 patch_prober.go:28] interesting pod/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 06 14:11:49.466519 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:49.466478 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" podUID="d38569a3-7eda-465b-be47-ddafa712a99d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 06 14:11:49.791113 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:49.791079 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:11:50.467088 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:50.467053 2577 patch_prober.go:28] interesting pod/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 06 14:11:50.467520 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:50.467108 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" podUID="d38569a3-7eda-465b-be47-ddafa712a99d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 06 14:11:51.212471 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:51.212427 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:51.627359 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:51.627322 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:51.628568 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:51.628533 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:52.318542 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.318514 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:52.343868 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.343831 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c"] Sep 06 14:11:52.344111 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.344083 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" podUID="995e8174-b715-4dd4-b913-e363072f6529" containerName="payload-processing" containerID="cri-o://2440369e2c194b4b002e7e30285226ea612024be66414640d59d99af881e7e41" gracePeriod=30 Sep 06 14:11:52.350601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.350571 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx"] Sep 06 14:11:52.350843 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.350820 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" podUID="0588afea-89c3-44cd-bbcb-05dc9a1ce41b" containerName="payload-pre-processing" containerID="cri-o://522ac21a1076af6c2a21873a1fe5a1d23e855d6bc7672e327ecac89ca8f360f1" gracePeriod=30 Sep 06 14:11:52.384846 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.384814 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:52.445080 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.445051 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:52.504681 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.504606 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:52.605681 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.605648 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" Sep 06 14:11:52.652932 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.652803 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" Sep 06 14:11:52.676861 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.676827 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv"] Sep 06 14:11:52.687740 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.687713 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:52.727813 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.727785 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/995e8174-b715-4dd4-b913-e363072f6529-plugins-config-volume\") pod \"995e8174-b715-4dd4-b913-e363072f6529\" (UID: \"995e8174-b715-4dd4-b913-e363072f6529\") " Sep 06 14:11:52.727984 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.727824 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llfm4\" (UniqueName: \"kubernetes.io/projected/0588afea-89c3-44cd-bbcb-05dc9a1ce41b-kube-api-access-llfm4\") pod \"0588afea-89c3-44cd-bbcb-05dc9a1ce41b\" (UID: \"0588afea-89c3-44cd-bbcb-05dc9a1ce41b\") " Sep 06 14:11:52.727984 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.727935 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0588afea-89c3-44cd-bbcb-05dc9a1ce41b-plugins-config-volume\") pod \"0588afea-89c3-44cd-bbcb-05dc9a1ce41b\" (UID: \"0588afea-89c3-44cd-bbcb-05dc9a1ce41b\") " Sep 06 14:11:52.727984 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.727972 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz8rm\" (UniqueName: \"kubernetes.io/projected/995e8174-b715-4dd4-b913-e363072f6529-kube-api-access-rz8rm\") pod \"995e8174-b715-4dd4-b913-e363072f6529\" (UID: \"995e8174-b715-4dd4-b913-e363072f6529\") " Sep 06 14:11:52.728259 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.728234 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995e8174-b715-4dd4-b913-e363072f6529-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "995e8174-b715-4dd4-b913-e363072f6529" (UID: "995e8174-b715-4dd4-b913-e363072f6529"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:11:52.728355 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.728317 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0588afea-89c3-44cd-bbcb-05dc9a1ce41b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0588afea-89c3-44cd-bbcb-05dc9a1ce41b" (UID: "0588afea-89c3-44cd-bbcb-05dc9a1ce41b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:11:52.728355 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.728332 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/995e8174-b715-4dd4-b913-e363072f6529-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:52.729865 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.729844 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0588afea-89c3-44cd-bbcb-05dc9a1ce41b-kube-api-access-llfm4" (OuterVolumeSpecName: "kube-api-access-llfm4") pod "0588afea-89c3-44cd-bbcb-05dc9a1ce41b" (UID: "0588afea-89c3-44cd-bbcb-05dc9a1ce41b"). InnerVolumeSpecName "kube-api-access-llfm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:11:52.729948 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.729918 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995e8174-b715-4dd4-b913-e363072f6529-kube-api-access-rz8rm" (OuterVolumeSpecName: "kube-api-access-rz8rm") pod "995e8174-b715-4dd4-b913-e363072f6529" (UID: "995e8174-b715-4dd4-b913-e363072f6529"). InnerVolumeSpecName "kube-api-access-rz8rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:11:52.829649 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.829608 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0588afea-89c3-44cd-bbcb-05dc9a1ce41b-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:52.829649 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.829640 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz8rm\" (UniqueName: \"kubernetes.io/projected/995e8174-b715-4dd4-b913-e363072f6529-kube-api-access-rz8rm\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:52.829649 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:52.829649 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llfm4\" (UniqueName: \"kubernetes.io/projected/0588afea-89c3-44cd-bbcb-05dc9a1ce41b-kube-api-access-llfm4\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:53.061862 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.061829 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:53.351062 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.350979 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:53.398383 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.398351 2577 generic.go:356] "Generic (PLEG): container finished" podID="0588afea-89c3-44cd-bbcb-05dc9a1ce41b" containerID="522ac21a1076af6c2a21873a1fe5a1d23e855d6bc7672e327ecac89ca8f360f1" exitCode=0 Sep 06 14:11:53.398565 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.398393 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" event={"ID":"0588afea-89c3-44cd-bbcb-05dc9a1ce41b","Type":"ContainerDied","Data":"522ac21a1076af6c2a21873a1fe5a1d23e855d6bc7672e327ecac89ca8f360f1"} Sep 06 14:11:53.398565 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.398472 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" event={"ID":"0588afea-89c3-44cd-bbcb-05dc9a1ce41b","Type":"ContainerDied","Data":"7b9c1c11b12d5336dfebed4b6febe27ffd6434540667a606cd30c86c218843ff"} Sep 06 14:11:53.398565 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.398405 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx" Sep 06 14:11:53.398565 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.398506 2577 scope.go:117] "RemoveContainer" containerID="522ac21a1076af6c2a21873a1fe5a1d23e855d6bc7672e327ecac89ca8f360f1" Sep 06 14:11:53.400147 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.400119 2577 generic.go:356] "Generic (PLEG): container finished" podID="995e8174-b715-4dd4-b913-e363072f6529" containerID="2440369e2c194b4b002e7e30285226ea612024be66414640d59d99af881e7e41" exitCode=0 Sep 06 14:11:53.400267 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.400175 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" event={"ID":"995e8174-b715-4dd4-b913-e363072f6529","Type":"ContainerDied","Data":"2440369e2c194b4b002e7e30285226ea612024be66414640d59d99af881e7e41"} Sep 06 14:11:53.400267 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.400189 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" Sep 06 14:11:53.400267 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.400229 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c" event={"ID":"995e8174-b715-4dd4-b913-e363072f6529","Type":"ContainerDied","Data":"a7f92c9c90a2c514ba2d6c2a0fed700d8d365e08ca1f89d88dadb071ad77b0af"} Sep 06 14:11:53.400457 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.400436 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" podUID="d38569a3-7eda-465b-be47-ddafa712a99d" containerName="istio-proxy" containerID="cri-o://895d36df299067fefbd8a15b44dc5acaaf2eb7ebae88b9b402ad735df3a6a6c4" gracePeriod=30 Sep 06 14:11:53.411073 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.411052 2577 scope.go:117] "RemoveContainer" containerID="522ac21a1076af6c2a21873a1fe5a1d23e855d6bc7672e327ecac89ca8f360f1" Sep 06 14:11:53.411428 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:11:53.411402 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522ac21a1076af6c2a21873a1fe5a1d23e855d6bc7672e327ecac89ca8f360f1\": container with ID starting with 522ac21a1076af6c2a21873a1fe5a1d23e855d6bc7672e327ecac89ca8f360f1 not found: ID does not exist" containerID="522ac21a1076af6c2a21873a1fe5a1d23e855d6bc7672e327ecac89ca8f360f1" Sep 06 14:11:53.411519 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.411429 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522ac21a1076af6c2a21873a1fe5a1d23e855d6bc7672e327ecac89ca8f360f1"} err="failed to get container status \"522ac21a1076af6c2a21873a1fe5a1d23e855d6bc7672e327ecac89ca8f360f1\": rpc error: code = NotFound desc = could not find container \"522ac21a1076af6c2a21873a1fe5a1d23e855d6bc7672e327ecac89ca8f360f1\": container with ID starting with 522ac21a1076af6c2a21873a1fe5a1d23e855d6bc7672e327ecac89ca8f360f1 not found: ID does not exist" Sep 06 14:11:53.411519 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.411448 2577 scope.go:117] "RemoveContainer" containerID="2440369e2c194b4b002e7e30285226ea612024be66414640d59d99af881e7e41" Sep 06 14:11:53.427824 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.427795 2577 scope.go:117] "RemoveContainer" containerID="2440369e2c194b4b002e7e30285226ea612024be66414640d59d99af881e7e41" Sep 06 14:11:53.428228 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:11:53.428201 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2440369e2c194b4b002e7e30285226ea612024be66414640d59d99af881e7e41\": container with ID starting with 2440369e2c194b4b002e7e30285226ea612024be66414640d59d99af881e7e41 not found: ID does not exist" containerID="2440369e2c194b4b002e7e30285226ea612024be66414640d59d99af881e7e41" Sep 06 14:11:53.428403 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.428373 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2440369e2c194b4b002e7e30285226ea612024be66414640d59d99af881e7e41"} err="failed to get container status \"2440369e2c194b4b002e7e30285226ea612024be66414640d59d99af881e7e41\": rpc error: code = NotFound desc = could not find container \"2440369e2c194b4b002e7e30285226ea612024be66414640d59d99af881e7e41\": container with ID starting with 2440369e2c194b4b002e7e30285226ea612024be66414640d59d99af881e7e41 not found: ID does not exist" Sep 06 14:11:53.442461 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.442430 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c"] Sep 06 14:11:53.445096 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.445072 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c0e08d9a-554745d694-8lz7c"] Sep 06 14:11:53.452412 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.452389 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx"] Sep 06 14:11:53.455958 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.455937 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c0e08d9a-7466c65bdb-fvjkx"] Sep 06 14:11:53.516676 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:53.516643 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:55.070193 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:55.070158 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0588afea-89c3-44cd-bbcb-05dc9a1ce41b" path="/var/lib/kubelet/pods/0588afea-89c3-44cd-bbcb-05dc9a1ce41b/volumes" Sep 06 14:11:55.070733 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:55.070714 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="995e8174-b715-4dd4-b913-e363072f6529" path="/var/lib/kubelet/pods/995e8174-b715-4dd4-b913-e363072f6529/volumes" Sep 06 14:11:55.473168 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:55.473081 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:55.575859 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:55.575832 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:56.684178 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:56.684145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:56.746607 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:56.746566 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:56.801353 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:56.801320 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:57.071054 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.071023 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:57.177373 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.177345 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:57.277926 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.277872 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:11:57.284550 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.284521 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr"] Sep 06 14:11:57.285043 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.285020 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0588afea-89c3-44cd-bbcb-05dc9a1ce41b" containerName="payload-pre-processing" Sep 06 14:11:57.285043 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.285041 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0588afea-89c3-44cd-bbcb-05dc9a1ce41b" containerName="payload-pre-processing" Sep 06 14:11:57.285202 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.285063 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="995e8174-b715-4dd4-b913-e363072f6529" containerName="payload-processing" Sep 06 14:11:57.285202 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.285069 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="995e8174-b715-4dd4-b913-e363072f6529" containerName="payload-processing" Sep 06 14:11:57.285202 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.285139 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="995e8174-b715-4dd4-b913-e363072f6529" containerName="payload-processing" Sep 06 14:11:57.285202 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.285149 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0588afea-89c3-44cd-bbcb-05dc9a1ce41b" containerName="payload-pre-processing" Sep 06 14:11:57.289996 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.289979 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" Sep 06 14:11:57.292702 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.292683 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d954782c-dockercfg-vjfjk\"" Sep 06 14:11:57.293011 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.292991 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d954782c\"" Sep 06 14:11:57.305725 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.305699 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr"] Sep 06 14:11:57.364863 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.364765 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z"] Sep 06 14:11:57.370792 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.370770 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" Sep 06 14:11:57.385475 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.385443 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z"] Sep 06 14:11:57.476215 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.476183 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752fa59e-9b79-40de-8abc-c309c2ddd129-plugins-config-volume\") pod \"payload-processing-e2e-ait-d954782c-75694cf88c-qth6z\" (UID: \"752fa59e-9b79-40de-8abc-c309c2ddd129\") " pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" Sep 06 14:11:57.476394 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.476235 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc223ca-c4db-41dd-be0e-34f03125e3ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr\" (UID: \"4bc223ca-c4db-41dd-be0e-34f03125e3ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" Sep 06 14:11:57.476394 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.476311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsnsc\" (UniqueName: \"kubernetes.io/projected/4bc223ca-c4db-41dd-be0e-34f03125e3ac-kube-api-access-gsnsc\") pod \"payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr\" (UID: \"4bc223ca-c4db-41dd-be0e-34f03125e3ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" Sep 06 14:11:57.476481 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.476410 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txkw6\" (UniqueName: \"kubernetes.io/projected/752fa59e-9b79-40de-8abc-c309c2ddd129-kube-api-access-txkw6\") pod \"payload-processing-e2e-ait-d954782c-75694cf88c-qth6z\" (UID: \"752fa59e-9b79-40de-8abc-c309c2ddd129\") " pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" Sep 06 14:11:57.577445 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.577411 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsnsc\" (UniqueName: \"kubernetes.io/projected/4bc223ca-c4db-41dd-be0e-34f03125e3ac-kube-api-access-gsnsc\") pod \"payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr\" (UID: \"4bc223ca-c4db-41dd-be0e-34f03125e3ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" Sep 06 14:11:57.577637 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.577478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txkw6\" (UniqueName: \"kubernetes.io/projected/752fa59e-9b79-40de-8abc-c309c2ddd129-kube-api-access-txkw6\") pod \"payload-processing-e2e-ait-d954782c-75694cf88c-qth6z\" (UID: \"752fa59e-9b79-40de-8abc-c309c2ddd129\") " pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" Sep 06 14:11:57.577637 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.577542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752fa59e-9b79-40de-8abc-c309c2ddd129-plugins-config-volume\") pod \"payload-processing-e2e-ait-d954782c-75694cf88c-qth6z\" (UID: \"752fa59e-9b79-40de-8abc-c309c2ddd129\") " pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" Sep 06 14:11:57.577637 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.577566 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc223ca-c4db-41dd-be0e-34f03125e3ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr\" (UID: \"4bc223ca-c4db-41dd-be0e-34f03125e3ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" Sep 06 14:11:57.578283 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.578265 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc223ca-c4db-41dd-be0e-34f03125e3ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr\" (UID: \"4bc223ca-c4db-41dd-be0e-34f03125e3ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" Sep 06 14:11:57.578332 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.578277 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752fa59e-9b79-40de-8abc-c309c2ddd129-plugins-config-volume\") pod \"payload-processing-e2e-ait-d954782c-75694cf88c-qth6z\" (UID: \"752fa59e-9b79-40de-8abc-c309c2ddd129\") " pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" Sep 06 14:11:57.585090 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.585061 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsnsc\" (UniqueName: \"kubernetes.io/projected/4bc223ca-c4db-41dd-be0e-34f03125e3ac-kube-api-access-gsnsc\") pod \"payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr\" (UID: \"4bc223ca-c4db-41dd-be0e-34f03125e3ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" Sep 06 14:11:57.585213 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.585112 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txkw6\" (UniqueName: \"kubernetes.io/projected/752fa59e-9b79-40de-8abc-c309c2ddd129-kube-api-access-txkw6\") pod \"payload-processing-e2e-ait-d954782c-75694cf88c-qth6z\" (UID: \"752fa59e-9b79-40de-8abc-c309c2ddd129\") " pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" Sep 06 14:11:57.601052 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.601022 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" Sep 06 14:11:57.687212 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.687186 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" Sep 06 14:11:57.880560 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.880531 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr"] Sep 06 14:11:57.882193 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:11:57.882167 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc223ca_c4db_41dd_be0e_34f03125e3ac.slice/crio-0e85161eb80903cad9c1b8aa14bf610db31f025bc3e77abae50f19c9d1d45ce7 WatchSource:0}: Error finding container 0e85161eb80903cad9c1b8aa14bf610db31f025bc3e77abae50f19c9d1d45ce7: Status 404 returned error can't find the container with id 0e85161eb80903cad9c1b8aa14bf610db31f025bc3e77abae50f19c9d1d45ce7 Sep 06 14:11:57.960342 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:57.960313 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z"] Sep 06 14:11:57.962679 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:11:57.962648 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752fa59e_9b79_40de_8abc_c309c2ddd129.slice/crio-63c1e52dd79ffda5e111bef3d853b94f7ed459ea332e407a1a9963858e905f06 WatchSource:0}: Error finding container 63c1e52dd79ffda5e111bef3d853b94f7ed459ea332e407a1a9963858e905f06: Status 404 returned error can't find the container with id 63c1e52dd79ffda5e111bef3d853b94f7ed459ea332e407a1a9963858e905f06 Sep 06 14:11:58.433229 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.433197 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" event={"ID":"752fa59e-9b79-40de-8abc-c309c2ddd129","Type":"ContainerStarted","Data":"b977c4a74607f1e3655452275240daa6ce04a9a9099da1fdb48a4c4bf3badc29"} Sep 06 14:11:58.433363 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.433257 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" event={"ID":"752fa59e-9b79-40de-8abc-c309c2ddd129","Type":"ContainerStarted","Data":"63c1e52dd79ffda5e111bef3d853b94f7ed459ea332e407a1a9963858e905f06"} Sep 06 14:11:58.433363 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.433302 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" Sep 06 14:11:58.435105 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.435081 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" event={"ID":"4bc223ca-c4db-41dd-be0e-34f03125e3ac","Type":"ContainerStarted","Data":"e8a7b8cb6d526b4f2871c661976546420ebca633326c136a28b1653255408f94"} Sep 06 14:11:58.435193 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.435125 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" event={"ID":"4bc223ca-c4db-41dd-be0e-34f03125e3ac","Type":"ContainerStarted","Data":"0e85161eb80903cad9c1b8aa14bf610db31f025bc3e77abae50f19c9d1d45ce7"} Sep 06 14:11:58.435193 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.435168 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" Sep 06 14:11:58.448185 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.448135 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" podStartSLOduration=1.44811679 podStartE2EDuration="1.44811679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:11:58.447112102 +0000 UTC m=+849.902600966" watchObservedRunningTime="2026-09-06 14:11:58.44811679 +0000 UTC m=+849.903605650" Sep 06 14:11:58.461580 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.461547 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" podStartSLOduration=1.461534089 podStartE2EDuration="1.461534089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:11:58.460122137 +0000 UTC m=+849.915610999" watchObservedRunningTime="2026-09-06 14:11:58.461534089 +0000 UTC m=+849.917022945" Sep 06 14:11:58.668354 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.668300 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:58.790819 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.790787 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-istio-data\") pod \"d38569a3-7eda-465b-be47-ddafa712a99d\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " Sep 06 14:11:58.791277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.790831 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wxjs\" (UniqueName: \"kubernetes.io/projected/d38569a3-7eda-465b-be47-ddafa712a99d-kube-api-access-2wxjs\") pod \"d38569a3-7eda-465b-be47-ddafa712a99d\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " Sep 06 14:11:58.791277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.790872 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-credential-socket\") pod \"d38569a3-7eda-465b-be47-ddafa712a99d\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " Sep 06 14:11:58.791277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.790902 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-workload-socket\") pod \"d38569a3-7eda-465b-be47-ddafa712a99d\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " Sep 06 14:11:58.791277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.790953 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d38569a3-7eda-465b-be47-ddafa712a99d-istiod-ca-cert\") pod \"d38569a3-7eda-465b-be47-ddafa712a99d\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " Sep 06 14:11:58.791277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.790984 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-workload-certs\") pod \"d38569a3-7eda-465b-be47-ddafa712a99d\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " Sep 06 14:11:58.791277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.791022 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d38569a3-7eda-465b-be47-ddafa712a99d-istio-token\") pod \"d38569a3-7eda-465b-be47-ddafa712a99d\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " Sep 06 14:11:58.791277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.791079 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-istio-envoy\") pod \"d38569a3-7eda-465b-be47-ddafa712a99d\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " Sep 06 14:11:58.791277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.791101 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d38569a3-7eda-465b-be47-ddafa712a99d-istio-podinfo\") pod \"d38569a3-7eda-465b-be47-ddafa712a99d\" (UID: \"d38569a3-7eda-465b-be47-ddafa712a99d\") " Sep 06 14:11:58.791277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.791117 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-istio-data" (OuterVolumeSpecName: "istio-data") pod "d38569a3-7eda-465b-be47-ddafa712a99d" (UID: "d38569a3-7eda-465b-be47-ddafa712a99d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:11:58.791277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.791188 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d38569a3-7eda-465b-be47-ddafa712a99d" (UID: "d38569a3-7eda-465b-be47-ddafa712a99d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:11:58.791277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.791232 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d38569a3-7eda-465b-be47-ddafa712a99d" (UID: "d38569a3-7eda-465b-be47-ddafa712a99d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:11:58.791820 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.791336 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d38569a3-7eda-465b-be47-ddafa712a99d" (UID: "d38569a3-7eda-465b-be47-ddafa712a99d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:11:58.791820 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.791348 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38569a3-7eda-465b-be47-ddafa712a99d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d38569a3-7eda-465b-be47-ddafa712a99d" (UID: "d38569a3-7eda-465b-be47-ddafa712a99d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:11:58.791820 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.791506 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-credential-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:58.791820 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.791525 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-workload-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:58.791820 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.791540 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d38569a3-7eda-465b-be47-ddafa712a99d-istiod-ca-cert\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:58.791820 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.791553 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-workload-certs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:58.791820 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.791569 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-istio-data\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:58.793274 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.793207 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38569a3-7eda-465b-be47-ddafa712a99d-kube-api-access-2wxjs" (OuterVolumeSpecName: "kube-api-access-2wxjs") pod "d38569a3-7eda-465b-be47-ddafa712a99d" (UID: "d38569a3-7eda-465b-be47-ddafa712a99d"). InnerVolumeSpecName "kube-api-access-2wxjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:11:58.793387 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.793266 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d38569a3-7eda-465b-be47-ddafa712a99d" (UID: "d38569a3-7eda-465b-be47-ddafa712a99d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:11:58.793566 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.793541 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38569a3-7eda-465b-be47-ddafa712a99d-istio-token" (OuterVolumeSpecName: "istio-token") pod "d38569a3-7eda-465b-be47-ddafa712a99d" (UID: "d38569a3-7eda-465b-be47-ddafa712a99d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:11:58.793670 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.793600 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d38569a3-7eda-465b-be47-ddafa712a99d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d38569a3-7eda-465b-be47-ddafa712a99d" (UID: "d38569a3-7eda-465b-be47-ddafa712a99d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 14:11:58.892363 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.892336 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d38569a3-7eda-465b-be47-ddafa712a99d-istio-token\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:58.892363 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.892360 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d38569a3-7eda-465b-be47-ddafa712a99d-istio-envoy\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:58.892528 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.892371 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d38569a3-7eda-465b-be47-ddafa712a99d-istio-podinfo\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:58.892528 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:58.892379 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wxjs\" (UniqueName: \"kubernetes.io/projected/d38569a3-7eda-465b-be47-ddafa712a99d-kube-api-access-2wxjs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:11:59.442157 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:59.442071 2577 generic.go:356] "Generic (PLEG): container finished" podID="d38569a3-7eda-465b-be47-ddafa712a99d" containerID="895d36df299067fefbd8a15b44dc5acaaf2eb7ebae88b9b402ad735df3a6a6c4" exitCode=0 Sep 06 14:11:59.442157 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:59.442115 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" Sep 06 14:11:59.442334 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:59.442150 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" event={"ID":"d38569a3-7eda-465b-be47-ddafa712a99d","Type":"ContainerDied","Data":"895d36df299067fefbd8a15b44dc5acaaf2eb7ebae88b9b402ad735df3a6a6c4"} Sep 06 14:11:59.442334 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:59.442206 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv" event={"ID":"d38569a3-7eda-465b-be47-ddafa712a99d","Type":"ContainerDied","Data":"4222ef1bd49d4379683a75416d7973a96fe490ef0be3708accfd108537d8c2f1"} Sep 06 14:11:59.442334 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:59.442230 2577 scope.go:117] "RemoveContainer" containerID="895d36df299067fefbd8a15b44dc5acaaf2eb7ebae88b9b402ad735df3a6a6c4" Sep 06 14:11:59.451373 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:59.451356 2577 scope.go:117] "RemoveContainer" containerID="3d361057493f3bf2143dbe87d4b944d1b3415575f451752a41fc1a657cdd6e8e" Sep 06 14:11:59.458713 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:59.458691 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv"] Sep 06 14:11:59.461555 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:59.461541 2577 scope.go:117] "RemoveContainer" containerID="895d36df299067fefbd8a15b44dc5acaaf2eb7ebae88b9b402ad735df3a6a6c4" Sep 06 14:11:59.461806 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:59.461789 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c0e08d9a-gw-openshift-default-74c8ccd776-p2mjv"] Sep 06 14:11:59.461867 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:11:59.461802 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"895d36df299067fefbd8a15b44dc5acaaf2eb7ebae88b9b402ad735df3a6a6c4\": container with ID starting with 895d36df299067fefbd8a15b44dc5acaaf2eb7ebae88b9b402ad735df3a6a6c4 not found: ID does not exist" containerID="895d36df299067fefbd8a15b44dc5acaaf2eb7ebae88b9b402ad735df3a6a6c4" Sep 06 14:11:59.461867 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:59.461826 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"895d36df299067fefbd8a15b44dc5acaaf2eb7ebae88b9b402ad735df3a6a6c4"} err="failed to get container status \"895d36df299067fefbd8a15b44dc5acaaf2eb7ebae88b9b402ad735df3a6a6c4\": rpc error: code = NotFound desc = could not find container \"895d36df299067fefbd8a15b44dc5acaaf2eb7ebae88b9b402ad735df3a6a6c4\": container with ID starting with 895d36df299067fefbd8a15b44dc5acaaf2eb7ebae88b9b402ad735df3a6a6c4 not found: ID does not exist" Sep 06 14:11:59.461867 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:59.461843 2577 scope.go:117] "RemoveContainer" containerID="3d361057493f3bf2143dbe87d4b944d1b3415575f451752a41fc1a657cdd6e8e" Sep 06 14:11:59.462120 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:11:59.462103 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d361057493f3bf2143dbe87d4b944d1b3415575f451752a41fc1a657cdd6e8e\": container with ID starting with 3d361057493f3bf2143dbe87d4b944d1b3415575f451752a41fc1a657cdd6e8e not found: ID does not exist" containerID="3d361057493f3bf2143dbe87d4b944d1b3415575f451752a41fc1a657cdd6e8e" Sep 06 14:11:59.462161 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:11:59.462124 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d361057493f3bf2143dbe87d4b944d1b3415575f451752a41fc1a657cdd6e8e"} err="failed to get container status \"3d361057493f3bf2143dbe87d4b944d1b3415575f451752a41fc1a657cdd6e8e\": rpc error: code = NotFound desc = could not find container \"3d361057493f3bf2143dbe87d4b944d1b3415575f451752a41fc1a657cdd6e8e\": container with ID starting with 3d361057493f3bf2143dbe87d4b944d1b3415575f451752a41fc1a657cdd6e8e not found: ID does not exist" Sep 06 14:12:01.068195 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:01.068159 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d38569a3-7eda-465b-be47-ddafa712a99d" path="/var/lib/kubelet/pods/d38569a3-7eda-465b-be47-ddafa712a99d/volumes" Sep 06 14:12:09.444843 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:09.444807 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" Sep 06 14:12:09.445363 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:09.444889 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" Sep 06 14:12:13.586842 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:13.586814 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:12:27.187508 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.187466 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:27.212252 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.212216 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z"] Sep 06 14:12:27.212480 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.212454 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" podUID="752fa59e-9b79-40de-8abc-c309c2ddd129" containerName="payload-processing" containerID="cri-o://b977c4a74607f1e3655452275240daa6ce04a9a9099da1fdb48a4c4bf3badc29" gracePeriod=30 Sep 06 14:12:27.217812 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.217786 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr"] Sep 06 14:12:27.218030 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.218007 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" podUID="4bc223ca-c4db-41dd-be0e-34f03125e3ac" containerName="payload-pre-processing" containerID="cri-o://e8a7b8cb6d526b4f2871c661976546420ebca633326c136a28b1653255408f94" gracePeriod=30 Sep 06 14:12:27.253736 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.253702 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:27.316619 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.316581 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:27.433181 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.431756 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:27.500719 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.500697 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" Sep 06 14:12:27.504130 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.504110 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" Sep 06 14:12:27.602783 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.602755 2577 generic.go:356] "Generic (PLEG): container finished" podID="752fa59e-9b79-40de-8abc-c309c2ddd129" containerID="b977c4a74607f1e3655452275240daa6ce04a9a9099da1fdb48a4c4bf3badc29" exitCode=0 Sep 06 14:12:27.602990 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.602795 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" Sep 06 14:12:27.602990 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.602825 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" event={"ID":"752fa59e-9b79-40de-8abc-c309c2ddd129","Type":"ContainerDied","Data":"b977c4a74607f1e3655452275240daa6ce04a9a9099da1fdb48a4c4bf3badc29"} Sep 06 14:12:27.602990 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.602873 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z" event={"ID":"752fa59e-9b79-40de-8abc-c309c2ddd129","Type":"ContainerDied","Data":"63c1e52dd79ffda5e111bef3d853b94f7ed459ea332e407a1a9963858e905f06"} Sep 06 14:12:27.602990 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.602893 2577 scope.go:117] "RemoveContainer" containerID="b977c4a74607f1e3655452275240daa6ce04a9a9099da1fdb48a4c4bf3badc29" Sep 06 14:12:27.604157 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.604138 2577 generic.go:356] "Generic (PLEG): container finished" podID="4bc223ca-c4db-41dd-be0e-34f03125e3ac" containerID="e8a7b8cb6d526b4f2871c661976546420ebca633326c136a28b1653255408f94" exitCode=0 Sep 06 14:12:27.604266 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.604199 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" Sep 06 14:12:27.604266 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.604222 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" event={"ID":"4bc223ca-c4db-41dd-be0e-34f03125e3ac","Type":"ContainerDied","Data":"e8a7b8cb6d526b4f2871c661976546420ebca633326c136a28b1653255408f94"} Sep 06 14:12:27.604266 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.604264 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr" event={"ID":"4bc223ca-c4db-41dd-be0e-34f03125e3ac","Type":"ContainerDied","Data":"0e85161eb80903cad9c1b8aa14bf610db31f025bc3e77abae50f19c9d1d45ce7"} Sep 06 14:12:27.612022 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.611997 2577 scope.go:117] "RemoveContainer" containerID="b977c4a74607f1e3655452275240daa6ce04a9a9099da1fdb48a4c4bf3badc29" Sep 06 14:12:27.612455 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:12:27.612431 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b977c4a74607f1e3655452275240daa6ce04a9a9099da1fdb48a4c4bf3badc29\": container with ID starting with b977c4a74607f1e3655452275240daa6ce04a9a9099da1fdb48a4c4bf3badc29 not found: ID does not exist" containerID="b977c4a74607f1e3655452275240daa6ce04a9a9099da1fdb48a4c4bf3badc29" Sep 06 14:12:27.612553 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.612464 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b977c4a74607f1e3655452275240daa6ce04a9a9099da1fdb48a4c4bf3badc29"} err="failed to get container status \"b977c4a74607f1e3655452275240daa6ce04a9a9099da1fdb48a4c4bf3badc29\": rpc error: code = NotFound desc = could not find container \"b977c4a74607f1e3655452275240daa6ce04a9a9099da1fdb48a4c4bf3badc29\": container with ID starting with b977c4a74607f1e3655452275240daa6ce04a9a9099da1fdb48a4c4bf3badc29 not found: ID does not exist" Sep 06 14:12:27.612553 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.612486 2577 scope.go:117] "RemoveContainer" containerID="e8a7b8cb6d526b4f2871c661976546420ebca633326c136a28b1653255408f94" Sep 06 14:12:27.621310 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.621294 2577 scope.go:117] "RemoveContainer" containerID="e8a7b8cb6d526b4f2871c661976546420ebca633326c136a28b1653255408f94" Sep 06 14:12:27.621599 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:12:27.621583 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a7b8cb6d526b4f2871c661976546420ebca633326c136a28b1653255408f94\": container with ID starting with e8a7b8cb6d526b4f2871c661976546420ebca633326c136a28b1653255408f94 not found: ID does not exist" containerID="e8a7b8cb6d526b4f2871c661976546420ebca633326c136a28b1653255408f94" Sep 06 14:12:27.621650 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.621605 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a7b8cb6d526b4f2871c661976546420ebca633326c136a28b1653255408f94"} err="failed to get container status \"e8a7b8cb6d526b4f2871c661976546420ebca633326c136a28b1653255408f94\": rpc error: code = NotFound desc = could not find container \"e8a7b8cb6d526b4f2871c661976546420ebca633326c136a28b1653255408f94\": container with ID starting with e8a7b8cb6d526b4f2871c661976546420ebca633326c136a28b1653255408f94 not found: ID does not exist" Sep 06 14:12:27.660309 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.658633 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsnsc\" (UniqueName: \"kubernetes.io/projected/4bc223ca-c4db-41dd-be0e-34f03125e3ac-kube-api-access-gsnsc\") pod \"4bc223ca-c4db-41dd-be0e-34f03125e3ac\" (UID: \"4bc223ca-c4db-41dd-be0e-34f03125e3ac\") " Sep 06 14:12:27.660309 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.658674 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txkw6\" (UniqueName: \"kubernetes.io/projected/752fa59e-9b79-40de-8abc-c309c2ddd129-kube-api-access-txkw6\") pod \"752fa59e-9b79-40de-8abc-c309c2ddd129\" (UID: \"752fa59e-9b79-40de-8abc-c309c2ddd129\") " Sep 06 14:12:27.660309 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.658843 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc223ca-c4db-41dd-be0e-34f03125e3ac-plugins-config-volume\") pod \"4bc223ca-c4db-41dd-be0e-34f03125e3ac\" (UID: \"4bc223ca-c4db-41dd-be0e-34f03125e3ac\") " Sep 06 14:12:27.660309 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.658877 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752fa59e-9b79-40de-8abc-c309c2ddd129-plugins-config-volume\") pod \"752fa59e-9b79-40de-8abc-c309c2ddd129\" (UID: \"752fa59e-9b79-40de-8abc-c309c2ddd129\") " Sep 06 14:12:27.660309 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.659430 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc223ca-c4db-41dd-be0e-34f03125e3ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4bc223ca-c4db-41dd-be0e-34f03125e3ac" (UID: "4bc223ca-c4db-41dd-be0e-34f03125e3ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:12:27.660309 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.659496 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752fa59e-9b79-40de-8abc-c309c2ddd129-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "752fa59e-9b79-40de-8abc-c309c2ddd129" (UID: "752fa59e-9b79-40de-8abc-c309c2ddd129"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:12:27.661496 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.661472 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc223ca-c4db-41dd-be0e-34f03125e3ac-kube-api-access-gsnsc" (OuterVolumeSpecName: "kube-api-access-gsnsc") pod "4bc223ca-c4db-41dd-be0e-34f03125e3ac" (UID: "4bc223ca-c4db-41dd-be0e-34f03125e3ac"). InnerVolumeSpecName "kube-api-access-gsnsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:12:27.661756 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.661737 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752fa59e-9b79-40de-8abc-c309c2ddd129-kube-api-access-txkw6" (OuterVolumeSpecName: "kube-api-access-txkw6") pod "752fa59e-9b79-40de-8abc-c309c2ddd129" (UID: "752fa59e-9b79-40de-8abc-c309c2ddd129"). InnerVolumeSpecName "kube-api-access-txkw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:12:27.759782 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.759731 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc223ca-c4db-41dd-be0e-34f03125e3ac-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:12:27.759782 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.759752 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752fa59e-9b79-40de-8abc-c309c2ddd129-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:12:27.759782 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.759761 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsnsc\" (UniqueName: \"kubernetes.io/projected/4bc223ca-c4db-41dd-be0e-34f03125e3ac-kube-api-access-gsnsc\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:12:27.759782 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.759769 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txkw6\" (UniqueName: \"kubernetes.io/projected/752fa59e-9b79-40de-8abc-c309c2ddd129-kube-api-access-txkw6\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:12:27.926135 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.926107 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr"] Sep 06 14:12:27.928294 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.928274 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d954782c-567689dcc8-7svdr"] Sep 06 14:12:27.937209 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.937159 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z"] Sep 06 14:12:27.938932 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:27.938896 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d954782c-75694cf88c-qth6z"] Sep 06 14:12:28.407462 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:28.407431 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:28.806602 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:28.806546 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:29.069615 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.069541 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bc223ca-c4db-41dd-be0e-34f03125e3ac" path="/var/lib/kubelet/pods/4bc223ca-c4db-41dd-be0e-34f03125e3ac/volumes" Sep 06 14:12:29.069985 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.069971 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="752fa59e-9b79-40de-8abc-c309c2ddd129" path="/var/lib/kubelet/pods/752fa59e-9b79-40de-8abc-c309c2ddd129/volumes" Sep 06 14:12:29.070233 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.070220 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl"] Sep 06 14:12:29.070638 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.070621 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bc223ca-c4db-41dd-be0e-34f03125e3ac" containerName="payload-pre-processing" Sep 06 14:12:29.070638 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.070637 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bc223ca-c4db-41dd-be0e-34f03125e3ac" containerName="payload-pre-processing" Sep 06 14:12:29.070785 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.070648 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d38569a3-7eda-465b-be47-ddafa712a99d" containerName="istio-proxy" Sep 06 14:12:29.070785 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.070654 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d38569a3-7eda-465b-be47-ddafa712a99d" containerName="istio-proxy" Sep 06 14:12:29.070785 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.070663 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="752fa59e-9b79-40de-8abc-c309c2ddd129" containerName="payload-processing" Sep 06 14:12:29.070785 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.070669 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="752fa59e-9b79-40de-8abc-c309c2ddd129" containerName="payload-processing" Sep 06 14:12:29.070785 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.070687 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d38569a3-7eda-465b-be47-ddafa712a99d" containerName="istio-proxy" Sep 06 14:12:29.070785 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.070692 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d38569a3-7eda-465b-be47-ddafa712a99d" containerName="istio-proxy" Sep 06 14:12:29.070785 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.070761 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bc223ca-c4db-41dd-be0e-34f03125e3ac" containerName="payload-pre-processing" Sep 06 14:12:29.070785 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.070770 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d38569a3-7eda-465b-be47-ddafa712a99d" containerName="istio-proxy" Sep 06 14:12:29.070785 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.070777 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d38569a3-7eda-465b-be47-ddafa712a99d" containerName="istio-proxy" Sep 06 14:12:29.070785 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.070783 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="752fa59e-9b79-40de-8abc-c309c2ddd129" containerName="payload-processing" Sep 06 14:12:29.075390 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.075366 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.077646 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.077619 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b1ed35f1-openshift-default-dockercfg-27g42\"" Sep 06 14:12:29.083248 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.082895 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl"] Sep 06 14:12:29.101291 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.101259 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:29.172451 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.172411 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-workload-certs\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.172612 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.172495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-workload-socket\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.172612 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.172533 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-token\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.172612 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.172578 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-data\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.172612 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.172609 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-envoy\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.172788 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.172628 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-podinfo\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.172788 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.172677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9wrs\" (UniqueName: \"kubernetes.io/projected/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-kube-api-access-n9wrs\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.172788 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.172725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istiod-ca-cert\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.172927 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.172799 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-credential-socket\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.273730 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.273694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9wrs\" (UniqueName: \"kubernetes.io/projected/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-kube-api-access-n9wrs\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.273730 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.273737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istiod-ca-cert\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.274018 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.273788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-credential-socket\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.274018 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.273827 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-workload-certs\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.274018 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.273943 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-workload-socket\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.274018 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.273986 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-token\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.274233 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.274024 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-data\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.274233 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.274060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-envoy\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.274233 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.274084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-podinfo\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.274389 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.274236 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-credential-socket\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.274389 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.274271 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-workload-certs\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.274389 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.274368 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-workload-socket\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.274555 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.274449 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-data\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.274624 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.274581 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istiod-ca-cert\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.276398 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.276367 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-envoy\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.276646 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.276625 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-podinfo\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.280858 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.280831 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9wrs\" (UniqueName: \"kubernetes.io/projected/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-kube-api-access-n9wrs\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.280858 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.280849 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-token\") pod \"e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.387146 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.387064 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:29.662945 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.662918 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl"] Sep 06 14:12:29.665417 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:12:29.665391 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bab5853_37f4_443f_aaf8_3dc5356c1eb4.slice/crio-c38e78856c97c2b81d7a868744745d1d2312bc82a3f36e6b68edb8715fced1c6 WatchSource:0}: Error finding container c38e78856c97c2b81d7a868744745d1d2312bc82a3f36e6b68edb8715fced1c6: Status 404 returned error can't find the container with id c38e78856c97c2b81d7a868744745d1d2312bc82a3f36e6b68edb8715fced1c6 Sep 06 14:12:29.667810 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.667774 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:12:29.667925 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.667856 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:12:29.667925 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.667886 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:12:29.684945 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:29.684922 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:30.037832 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:30.037766 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:30.623173 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:30.623135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" event={"ID":"7bab5853-37f4-443f-aaf8-3dc5356c1eb4","Type":"ContainerStarted","Data":"9efcb5e249566afcf62b116c2dfdb10469a098ead3305cb04028ab50a16a0e50"} Sep 06 14:12:30.623361 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:30.623198 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" event={"ID":"7bab5853-37f4-443f-aaf8-3dc5356c1eb4","Type":"ContainerStarted","Data":"c38e78856c97c2b81d7a868744745d1d2312bc82a3f36e6b68edb8715fced1c6"} Sep 06 14:12:30.640783 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:30.640732 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" podStartSLOduration=1.6407156170000001 podStartE2EDuration="1.640715617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:12:30.639800418 +0000 UTC m=+882.095289272" watchObservedRunningTime="2026-09-06 14:12:30.640715617 +0000 UTC m=+882.096204478" Sep 06 14:12:30.783360 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:30.783328 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:31.042860 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:31.042828 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf"] Sep 06 14:12:31.046560 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:31.046540 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf" Sep 06 14:12:31.049121 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:31.049099 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\"" Sep 06 14:12:31.049549 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:31.049280 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 14:12:31.051732 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:31.051699 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf"] Sep 06 14:12:31.193861 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:31.193824 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b169e109-e833-4fff-a5a8-97c1d8125c9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-1482ae-wmhtf\" (UID: \"b169e109-e833-4fff-a5a8-97c1d8125c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf" Sep 06 14:12:31.221606 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:31.221570 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:31.294807 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:31.294720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b169e109-e833-4fff-a5a8-97c1d8125c9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-1482ae-wmhtf\" (UID: \"b169e109-e833-4fff-a5a8-97c1d8125c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf" Sep 06 14:12:31.295384 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:31.295364 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b169e109-e833-4fff-a5a8-97c1d8125c9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-1482ae-wmhtf\" (UID: \"b169e109-e833-4fff-a5a8-97c1d8125c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf" Sep 06 14:12:31.362358 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:31.362331 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf" Sep 06 14:12:31.387284 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:31.387261 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:31.393834 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:31.393809 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:31.629014 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:31.628943 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:31.630345 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:31.630322 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:31.633148 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:31.633128 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf"] Sep 06 14:12:31.635554 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:12:31.635530 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb169e109_e833_4fff_a5a8_97c1d8125c9f.slice/crio-220a2547a7307d15108eb60b37852430b5a43d86720da0d880e2fb445b3c7b17 WatchSource:0}: Error finding container 220a2547a7307d15108eb60b37852430b5a43d86720da0d880e2fb445b3c7b17: Status 404 returned error can't find the container with id 220a2547a7307d15108eb60b37852430b5a43d86720da0d880e2fb445b3c7b17 Sep 06 14:12:32.053265 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:32.053232 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:32.149433 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:32.149400 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:32.635503 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:32.635468 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf" event={"ID":"b169e109-e833-4fff-a5a8-97c1d8125c9f","Type":"ContainerStarted","Data":"220a2547a7307d15108eb60b37852430b5a43d86720da0d880e2fb445b3c7b17"} Sep 06 14:12:33.220016 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:33.219972 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:33.318647 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:33.318615 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:33.494231 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:33.494148 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:33.773658 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:33.773623 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:33.872286 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:33.872221 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:33.873260 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:33.873238 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll"] Sep 06 14:12:33.882964 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:33.882940 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" Sep 06 14:12:33.885357 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:33.885335 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b1ed35f1\"" Sep 06 14:12:33.885673 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:33.885335 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b1ed35f1-dockercfg-klw7c\"" Sep 06 14:12:33.886380 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:33.886357 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll"] Sep 06 14:12:33.917387 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:33.917358 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr"] Sep 06 14:12:33.922442 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:33.922419 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" Sep 06 14:12:33.928361 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:33.928336 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr"] Sep 06 14:12:34.021018 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.020984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bdf53fd-6ef7-4494-b778-546b7372e8a3-plugins-config-volume\") pod \"payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr\" (UID: \"2bdf53fd-6ef7-4494-b778-546b7372e8a3\") " pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" Sep 06 14:12:34.021204 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.021033 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpjr7\" (UniqueName: \"kubernetes.io/projected/27d64c34-6f87-45cc-89b3-665590b87530-kube-api-access-qpjr7\") pod \"payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll\" (UID: \"27d64c34-6f87-45cc-89b3-665590b87530\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" Sep 06 14:12:34.021311 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.021194 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d64c34-6f87-45cc-89b3-665590b87530-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll\" (UID: \"27d64c34-6f87-45cc-89b3-665590b87530\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" Sep 06 14:12:34.021311 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.021238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcf66\" (UniqueName: \"kubernetes.io/projected/2bdf53fd-6ef7-4494-b778-546b7372e8a3-kube-api-access-rcf66\") pod \"payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr\" (UID: \"2bdf53fd-6ef7-4494-b778-546b7372e8a3\") " pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" Sep 06 14:12:34.100089 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.100007 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:34.122250 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.122218 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpjr7\" (UniqueName: \"kubernetes.io/projected/27d64c34-6f87-45cc-89b3-665590b87530-kube-api-access-qpjr7\") pod \"payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll\" (UID: \"27d64c34-6f87-45cc-89b3-665590b87530\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" Sep 06 14:12:34.122408 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.122333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d64c34-6f87-45cc-89b3-665590b87530-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll\" (UID: \"27d64c34-6f87-45cc-89b3-665590b87530\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" Sep 06 14:12:34.122408 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.122364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcf66\" (UniqueName: \"kubernetes.io/projected/2bdf53fd-6ef7-4494-b778-546b7372e8a3-kube-api-access-rcf66\") pod \"payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr\" (UID: \"2bdf53fd-6ef7-4494-b778-546b7372e8a3\") " pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" Sep 06 14:12:34.122510 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.122451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bdf53fd-6ef7-4494-b778-546b7372e8a3-plugins-config-volume\") pod \"payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr\" (UID: \"2bdf53fd-6ef7-4494-b778-546b7372e8a3\") " pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" Sep 06 14:12:34.123314 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.123282 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bdf53fd-6ef7-4494-b778-546b7372e8a3-plugins-config-volume\") pod \"payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr\" (UID: \"2bdf53fd-6ef7-4494-b778-546b7372e8a3\") " pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" Sep 06 14:12:34.123389 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.123290 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d64c34-6f87-45cc-89b3-665590b87530-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll\" (UID: \"27d64c34-6f87-45cc-89b3-665590b87530\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" Sep 06 14:12:34.131826 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.131799 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpjr7\" (UniqueName: \"kubernetes.io/projected/27d64c34-6f87-45cc-89b3-665590b87530-kube-api-access-qpjr7\") pod \"payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll\" (UID: \"27d64c34-6f87-45cc-89b3-665590b87530\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" Sep 06 14:12:34.131952 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.131834 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcf66\" (UniqueName: \"kubernetes.io/projected/2bdf53fd-6ef7-4494-b778-546b7372e8a3-kube-api-access-rcf66\") pod \"payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr\" (UID: \"2bdf53fd-6ef7-4494-b778-546b7372e8a3\") " pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" Sep 06 14:12:34.197768 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.197733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" Sep 06 14:12:34.235031 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.235003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" Sep 06 14:12:34.300874 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.300839 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:12:34.534702 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.534671 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll"] Sep 06 14:12:34.536956 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.536935 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr"] Sep 06 14:12:34.538068 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:12:34.538044 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d64c34_6f87_45cc_89b3_665590b87530.slice/crio-4ea0a04d6516485e63336175a063f8fddda81d8591da1e827b54d6e2eb4f55d4 WatchSource:0}: Error finding container 4ea0a04d6516485e63336175a063f8fddda81d8591da1e827b54d6e2eb4f55d4: Status 404 returned error can't find the container with id 4ea0a04d6516485e63336175a063f8fddda81d8591da1e827b54d6e2eb4f55d4 Sep 06 14:12:34.539053 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:12:34.539000 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdf53fd_6ef7_4494_b778_546b7372e8a3.slice/crio-43be55b06996314f231ccf77af0b10007861d49e999ab13464936b75a6ec31f8 WatchSource:0}: Error finding container 43be55b06996314f231ccf77af0b10007861d49e999ab13464936b75a6ec31f8: Status 404 returned error can't find the container with id 43be55b06996314f231ccf77af0b10007861d49e999ab13464936b75a6ec31f8 Sep 06 14:12:34.649491 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.649457 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" event={"ID":"27d64c34-6f87-45cc-89b3-665590b87530","Type":"ContainerStarted","Data":"ce53105e917540d05c6498ff2921e9df1bda0d817f5ac48a282bab2e227669e7"} Sep 06 14:12:34.649624 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.649523 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" event={"ID":"27d64c34-6f87-45cc-89b3-665590b87530","Type":"ContainerStarted","Data":"4ea0a04d6516485e63336175a063f8fddda81d8591da1e827b54d6e2eb4f55d4"} Sep 06 14:12:34.649624 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.649557 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" Sep 06 14:12:34.651248 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.651221 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" event={"ID":"2bdf53fd-6ef7-4494-b778-546b7372e8a3","Type":"ContainerStarted","Data":"30423936cbebc52ff506e2e30876456e309f01a25cdde7c35a87d12af163210e"} Sep 06 14:12:34.651388 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.651273 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" event={"ID":"2bdf53fd-6ef7-4494-b778-546b7372e8a3","Type":"ContainerStarted","Data":"43be55b06996314f231ccf77af0b10007861d49e999ab13464936b75a6ec31f8"} Sep 06 14:12:34.651388 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.651335 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" Sep 06 14:12:34.652959 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.652931 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf" event={"ID":"b169e109-e833-4fff-a5a8-97c1d8125c9f","Type":"ContainerStarted","Data":"4da7e1c6fd4a83cecb0a0edd8b8c9328d383521b763f77f4105022d6c9a28a54"} Sep 06 14:12:34.680708 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.680648 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" podStartSLOduration=1.6806311 podStartE2EDuration="1.6806311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:12:34.664164796 +0000 UTC m=+886.119653659" watchObservedRunningTime="2026-09-06 14:12:34.6806311 +0000 UTC m=+886.136119955" Sep 06 14:12:34.681134 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.681106 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" podStartSLOduration=1.681095904 podStartE2EDuration="1.681095904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:12:34.680121023 +0000 UTC m=+886.135609885" watchObservedRunningTime="2026-09-06 14:12:34.681095904 +0000 UTC m=+886.136584766" Sep 06 14:12:34.692989 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:34.692939 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf" podStartSLOduration=1.001331856 podStartE2EDuration="3.692893101s" totalImagesPullingTime="2.691561245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:12:34.692645685 +0000 UTC m=+886.148134550" watchObservedRunningTime="2026-09-06 14:12:34.692893101 +0000 UTC m=+886.148381964" Sep 06 14:12:35.661633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:35.661598 2577 generic.go:356] "Generic (PLEG): container finished" podID="b169e109-e833-4fff-a5a8-97c1d8125c9f" containerID="4da7e1c6fd4a83cecb0a0edd8b8c9328d383521b763f77f4105022d6c9a28a54" exitCode=0 Sep 06 14:12:35.662083 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:35.661649 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf" event={"ID":"b169e109-e833-4fff-a5a8-97c1d8125c9f","Type":"ContainerDied","Data":"4da7e1c6fd4a83cecb0a0edd8b8c9328d383521b763f77f4105022d6c9a28a54"} Sep 06 14:12:36.800947 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:36.800924 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf" Sep 06 14:12:36.951284 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:36.951217 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b169e109-e833-4fff-a5a8-97c1d8125c9f-maas-api-service-ca\") pod \"b169e109-e833-4fff-a5a8-97c1d8125c9f\" (UID: \"b169e109-e833-4fff-a5a8-97c1d8125c9f\") " Sep 06 14:12:36.951588 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:36.951567 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b169e109-e833-4fff-a5a8-97c1d8125c9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b169e109-e833-4fff-a5a8-97c1d8125c9f" (UID: "b169e109-e833-4fff-a5a8-97c1d8125c9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:12:37.052323 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:37.052295 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b169e109-e833-4fff-a5a8-97c1d8125c9f-maas-api-service-ca\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:12:37.674404 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:37.674369 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf" event={"ID":"b169e109-e833-4fff-a5a8-97c1d8125c9f","Type":"ContainerDied","Data":"220a2547a7307d15108eb60b37852430b5a43d86720da0d880e2fb445b3c7b17"} Sep 06 14:12:37.674404 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:37.674388 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf" Sep 06 14:12:37.674658 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:37.674421 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="220a2547a7307d15108eb60b37852430b5a43d86720da0d880e2fb445b3c7b17" Sep 06 14:12:37.892345 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:37.892314 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:40.200474 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:40.200413 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:41.213729 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:41.213657 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:41.811882 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:41.811854 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:41.837805 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:41.837745 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn"] Sep 06 14:12:41.838570 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:41.838537 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" podUID="64788558-c059-4cda-a753-64d3393bfd88" containerName="payload-processing" containerID="cri-o://93232882a31dad62c1bbbceec4997e3d7d43c2c54e937abe0247e4407c47dc2d" gracePeriod=30 Sep 06 14:12:41.841368 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:41.841029 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64"] Sep 06 14:12:41.841496 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:41.841467 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" podUID="017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9" containerName="payload-pre-processing" containerID="cri-o://94a64491cb6311ec0d964202743f7638ca4682939974f567ea0ba6c6da0dafef" gracePeriod=30 Sep 06 14:12:41.878521 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:41.878500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:41.975504 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:41.975460 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:41.984637 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:41.984612 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf"] Sep 06 14:12:41.988061 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:41.988030 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1482ae-wmhtf"] Sep 06 14:12:42.002546 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.002514 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" podUID="017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.37:9004: connect: connection refused" Sep 06 14:12:42.002671 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.002561 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" podUID="64788558-c059-4cda-a753-64d3393bfd88" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.38:9004: connect: connection refused" Sep 06 14:12:42.077625 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.077512 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:42.130789 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.130769 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" Sep 06 14:12:42.133767 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.133751 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" Sep 06 14:12:42.301090 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.301062 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75s2f\" (UniqueName: \"kubernetes.io/projected/017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9-kube-api-access-75s2f\") pod \"017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9\" (UID: \"017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9\") " Sep 06 14:12:42.301440 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.301158 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64788558-c059-4cda-a753-64d3393bfd88-plugins-config-volume\") pod \"64788558-c059-4cda-a753-64d3393bfd88\" (UID: \"64788558-c059-4cda-a753-64d3393bfd88\") " Sep 06 14:12:42.301440 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.301179 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl45z\" (UniqueName: \"kubernetes.io/projected/64788558-c059-4cda-a753-64d3393bfd88-kube-api-access-rl45z\") pod \"64788558-c059-4cda-a753-64d3393bfd88\" (UID: \"64788558-c059-4cda-a753-64d3393bfd88\") " Sep 06 14:12:42.301440 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.301211 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9-plugins-config-volume\") pod \"017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9\" (UID: \"017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9\") " Sep 06 14:12:42.301600 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.301570 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64788558-c059-4cda-a753-64d3393bfd88-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64788558-c059-4cda-a753-64d3393bfd88" (UID: "64788558-c059-4cda-a753-64d3393bfd88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:12:42.301645 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.301613 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9" (UID: "017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:12:42.303122 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.303106 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9-kube-api-access-75s2f" (OuterVolumeSpecName: "kube-api-access-75s2f") pod "017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9" (UID: "017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9"). InnerVolumeSpecName "kube-api-access-75s2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:12:42.303254 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.303233 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64788558-c059-4cda-a753-64d3393bfd88-kube-api-access-rl45z" (OuterVolumeSpecName: "kube-api-access-rl45z") pod "64788558-c059-4cda-a753-64d3393bfd88" (UID: "64788558-c059-4cda-a753-64d3393bfd88"). InnerVolumeSpecName "kube-api-access-rl45z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:12:42.313793 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.313768 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:42.402702 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.402645 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:12:42.402702 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.402667 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75s2f\" (UniqueName: \"kubernetes.io/projected/017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9-kube-api-access-75s2f\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:12:42.402702 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.402678 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64788558-c059-4cda-a753-64d3393bfd88-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:12:42.402702 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.402687 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl45z\" (UniqueName: \"kubernetes.io/projected/64788558-c059-4cda-a753-64d3393bfd88-kube-api-access-rl45z\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:12:42.701366 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.701281 2577 generic.go:356] "Generic (PLEG): container finished" podID="017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9" containerID="94a64491cb6311ec0d964202743f7638ca4682939974f567ea0ba6c6da0dafef" exitCode=0 Sep 06 14:12:42.701366 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.701330 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" Sep 06 14:12:42.701561 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.701370 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" event={"ID":"017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9","Type":"ContainerDied","Data":"94a64491cb6311ec0d964202743f7638ca4682939974f567ea0ba6c6da0dafef"} Sep 06 14:12:42.701561 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.701420 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64" event={"ID":"017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9","Type":"ContainerDied","Data":"21b9d6fdaf18d6b5b8fc2fadc68b8475975f88b746721bc8804e99ec90340bc3"} Sep 06 14:12:42.701561 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.701441 2577 scope.go:117] "RemoveContainer" containerID="94a64491cb6311ec0d964202743f7638ca4682939974f567ea0ba6c6da0dafef" Sep 06 14:12:42.702812 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.702785 2577 generic.go:356] "Generic (PLEG): container finished" podID="64788558-c059-4cda-a753-64d3393bfd88" containerID="93232882a31dad62c1bbbceec4997e3d7d43c2c54e937abe0247e4407c47dc2d" exitCode=0 Sep 06 14:12:42.702929 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.702810 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" event={"ID":"64788558-c059-4cda-a753-64d3393bfd88","Type":"ContainerDied","Data":"93232882a31dad62c1bbbceec4997e3d7d43c2c54e937abe0247e4407c47dc2d"} Sep 06 14:12:42.702929 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.702828 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" Sep 06 14:12:42.702929 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.702854 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn" event={"ID":"64788558-c059-4cda-a753-64d3393bfd88","Type":"ContainerDied","Data":"e6b56798002fa117423da05068fe3b9b3bcf2289a416770b27f30b448bae2ccf"} Sep 06 14:12:42.711540 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.711524 2577 scope.go:117] "RemoveContainer" containerID="94a64491cb6311ec0d964202743f7638ca4682939974f567ea0ba6c6da0dafef" Sep 06 14:12:42.712006 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:12:42.711795 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a64491cb6311ec0d964202743f7638ca4682939974f567ea0ba6c6da0dafef\": container with ID starting with 94a64491cb6311ec0d964202743f7638ca4682939974f567ea0ba6c6da0dafef not found: ID does not exist" containerID="94a64491cb6311ec0d964202743f7638ca4682939974f567ea0ba6c6da0dafef" Sep 06 14:12:42.712006 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.711828 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a64491cb6311ec0d964202743f7638ca4682939974f567ea0ba6c6da0dafef"} err="failed to get container status \"94a64491cb6311ec0d964202743f7638ca4682939974f567ea0ba6c6da0dafef\": rpc error: code = NotFound desc = could not find container \"94a64491cb6311ec0d964202743f7638ca4682939974f567ea0ba6c6da0dafef\": container with ID starting with 94a64491cb6311ec0d964202743f7638ca4682939974f567ea0ba6c6da0dafef not found: ID does not exist" Sep 06 14:12:42.712006 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.711846 2577 scope.go:117] "RemoveContainer" containerID="93232882a31dad62c1bbbceec4997e3d7d43c2c54e937abe0247e4407c47dc2d" Sep 06 14:12:42.719983 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.719961 2577 scope.go:117] "RemoveContainer" containerID="93232882a31dad62c1bbbceec4997e3d7d43c2c54e937abe0247e4407c47dc2d" Sep 06 14:12:42.720236 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:12:42.720218 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93232882a31dad62c1bbbceec4997e3d7d43c2c54e937abe0247e4407c47dc2d\": container with ID starting with 93232882a31dad62c1bbbceec4997e3d7d43c2c54e937abe0247e4407c47dc2d not found: ID does not exist" containerID="93232882a31dad62c1bbbceec4997e3d7d43c2c54e937abe0247e4407c47dc2d" Sep 06 14:12:42.720304 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.720242 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93232882a31dad62c1bbbceec4997e3d7d43c2c54e937abe0247e4407c47dc2d"} err="failed to get container status \"93232882a31dad62c1bbbceec4997e3d7d43c2c54e937abe0247e4407c47dc2d\": rpc error: code = NotFound desc = could not find container \"93232882a31dad62c1bbbceec4997e3d7d43c2c54e937abe0247e4407c47dc2d\": container with ID starting with 93232882a31dad62c1bbbceec4997e3d7d43c2c54e937abe0247e4407c47dc2d not found: ID does not exist" Sep 06 14:12:42.727942 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.727922 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn"] Sep 06 14:12:42.730916 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.730882 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1482ae-c97b58f87-mz5gn"] Sep 06 14:12:42.739833 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.739810 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64"] Sep 06 14:12:42.743213 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:42.743194 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1482ae-64d8d45554-nxd64"] Sep 06 14:12:43.069691 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:43.069664 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9" path="/var/lib/kubelet/pods/017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9/volumes" Sep 06 14:12:43.070055 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:43.070043 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64788558-c059-4cda-a753-64d3393bfd88" path="/var/lib/kubelet/pods/64788558-c059-4cda-a753-64d3393bfd88/volumes" Sep 06 14:12:43.070334 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:43.070323 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b169e109-e833-4fff-a5a8-97c1d8125c9f" path="/var/lib/kubelet/pods/b169e109-e833-4fff-a5a8-97c1d8125c9f/volumes" Sep 06 14:12:45.664038 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:45.664011 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" Sep 06 14:12:45.664470 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:45.664071 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" Sep 06 14:12:47.390248 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:47.390210 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:54.987548 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:54.987517 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:12:55.211216 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:55.211167 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:56.792783 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:56.792746 2577 generic.go:356] "Generic (PLEG): container finished" podID="7bab5853-37f4-443f-aaf8-3dc5356c1eb4" containerID="9efcb5e249566afcf62b116c2dfdb10469a098ead3305cb04028ab50a16a0e50" exitCode=0 Sep 06 14:12:56.793195 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:56.792821 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" event={"ID":"7bab5853-37f4-443f-aaf8-3dc5356c1eb4","Type":"ContainerDied","Data":"9efcb5e249566afcf62b116c2dfdb10469a098ead3305cb04028ab50a16a0e50"} Sep 06 14:12:56.793241 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:56.793210 2577 scope.go:117] "RemoveContainer" containerID="9efcb5e249566afcf62b116c2dfdb10469a098ead3305cb04028ab50a16a0e50" Sep 06 14:12:56.794259 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:56.794236 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:12:56.794325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:56.794303 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:12:56.794371 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:56.794337 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:12:57.439398 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:57.439350 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:12:57.799607 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:57.799577 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" event={"ID":"7bab5853-37f4-443f-aaf8-3dc5356c1eb4","Type":"ContainerStarted","Data":"efefe9e9f8aaf98186a3841bfc5843927dba1895eb321586b4a04ace47163c36"} Sep 06 14:12:58.387283 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:58.387250 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:12:58.388373 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:58.388344 2577 patch_prober.go:28] interesting pod/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 06 14:12:58.388505 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:58.388394 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" podUID="7bab5853-37f4-443f-aaf8-3dc5356c1eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 06 14:12:59.387710 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:59.387679 2577 patch_prober.go:28] interesting pod/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 06 14:12:59.388119 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:12:59.387729 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" podUID="7bab5853-37f4-443f-aaf8-3dc5356c1eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 06 14:13:00.387671 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:00.387635 2577 patch_prober.go:28] interesting pod/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 06 14:13:00.387855 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:00.387698 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" podUID="7bab5853-37f4-443f-aaf8-3dc5356c1eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 06 14:13:01.388073 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:01.388033 2577 patch_prober.go:28] interesting pod/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 06 14:13:01.388450 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:01.388095 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" podUID="7bab5853-37f4-443f-aaf8-3dc5356c1eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 06 14:13:01.629689 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:01.629653 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:13:02.388189 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:02.388148 2577 patch_prober.go:28] interesting pod/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 06 14:13:02.388704 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:02.388222 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" podUID="7bab5853-37f4-443f-aaf8-3dc5356c1eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 06 14:13:03.392070 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:03.392038 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:13:03.393066 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:03.393048 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:13:03.744265 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:03.744184 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:03.769267 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:03.769233 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr"] Sep 06 14:13:03.769495 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:03.769469 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" podUID="2bdf53fd-6ef7-4494-b778-546b7372e8a3" containerName="payload-processing" containerID="cri-o://30423936cbebc52ff506e2e30876456e309f01a25cdde7c35a87d12af163210e" gracePeriod=30 Sep 06 14:13:03.778310 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:03.778272 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll"] Sep 06 14:13:03.778506 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:03.778477 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" podUID="27d64c34-6f87-45cc-89b3-665590b87530" containerName="payload-pre-processing" containerID="cri-o://ce53105e917540d05c6498ff2921e9df1bda0d817f5ac48a282bab2e227669e7" gracePeriod=30 Sep 06 14:13:04.048629 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.048520 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl"] Sep 06 14:13:04.058671 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.058639 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" Sep 06 14:13:04.079566 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.079541 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" Sep 06 14:13:04.110083 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.110056 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bdf53fd-6ef7-4494-b778-546b7372e8a3-plugins-config-volume\") pod \"2bdf53fd-6ef7-4494-b778-546b7372e8a3\" (UID: \"2bdf53fd-6ef7-4494-b778-546b7372e8a3\") " Sep 06 14:13:04.110235 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.110202 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcf66\" (UniqueName: \"kubernetes.io/projected/2bdf53fd-6ef7-4494-b778-546b7372e8a3-kube-api-access-rcf66\") pod \"2bdf53fd-6ef7-4494-b778-546b7372e8a3\" (UID: \"2bdf53fd-6ef7-4494-b778-546b7372e8a3\") " Sep 06 14:13:04.110459 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.110430 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdf53fd-6ef7-4494-b778-546b7372e8a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2bdf53fd-6ef7-4494-b778-546b7372e8a3" (UID: "2bdf53fd-6ef7-4494-b778-546b7372e8a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:13:04.110775 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.110752 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bdf53fd-6ef7-4494-b778-546b7372e8a3-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:13:04.112400 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.112380 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdf53fd-6ef7-4494-b778-546b7372e8a3-kube-api-access-rcf66" (OuterVolumeSpecName: "kube-api-access-rcf66") pod "2bdf53fd-6ef7-4494-b778-546b7372e8a3" (UID: "2bdf53fd-6ef7-4494-b778-546b7372e8a3"). InnerVolumeSpecName "kube-api-access-rcf66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:13:04.211623 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.211588 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpjr7\" (UniqueName: \"kubernetes.io/projected/27d64c34-6f87-45cc-89b3-665590b87530-kube-api-access-qpjr7\") pod \"27d64c34-6f87-45cc-89b3-665590b87530\" (UID: \"27d64c34-6f87-45cc-89b3-665590b87530\") " Sep 06 14:13:04.211787 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.211646 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d64c34-6f87-45cc-89b3-665590b87530-plugins-config-volume\") pod \"27d64c34-6f87-45cc-89b3-665590b87530\" (UID: \"27d64c34-6f87-45cc-89b3-665590b87530\") " Sep 06 14:13:04.211872 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.211855 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcf66\" (UniqueName: \"kubernetes.io/projected/2bdf53fd-6ef7-4494-b778-546b7372e8a3-kube-api-access-rcf66\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:13:04.212072 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.212053 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d64c34-6f87-45cc-89b3-665590b87530-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27d64c34-6f87-45cc-89b3-665590b87530" (UID: "27d64c34-6f87-45cc-89b3-665590b87530"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:13:04.213614 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.213592 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d64c34-6f87-45cc-89b3-665590b87530-kube-api-access-qpjr7" (OuterVolumeSpecName: "kube-api-access-qpjr7") pod "27d64c34-6f87-45cc-89b3-665590b87530" (UID: "27d64c34-6f87-45cc-89b3-665590b87530"). InnerVolumeSpecName "kube-api-access-qpjr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:13:04.312800 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.312768 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpjr7\" (UniqueName: \"kubernetes.io/projected/27d64c34-6f87-45cc-89b3-665590b87530-kube-api-access-qpjr7\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:13:04.312800 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.312796 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d64c34-6f87-45cc-89b3-665590b87530-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:13:04.838965 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.838929 2577 generic.go:356] "Generic (PLEG): container finished" podID="2bdf53fd-6ef7-4494-b778-546b7372e8a3" containerID="30423936cbebc52ff506e2e30876456e309f01a25cdde7c35a87d12af163210e" exitCode=0 Sep 06 14:13:04.839393 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.838974 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" Sep 06 14:13:04.839393 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.839003 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" event={"ID":"2bdf53fd-6ef7-4494-b778-546b7372e8a3","Type":"ContainerDied","Data":"30423936cbebc52ff506e2e30876456e309f01a25cdde7c35a87d12af163210e"} Sep 06 14:13:04.839393 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.839056 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr" event={"ID":"2bdf53fd-6ef7-4494-b778-546b7372e8a3","Type":"ContainerDied","Data":"43be55b06996314f231ccf77af0b10007861d49e999ab13464936b75a6ec31f8"} Sep 06 14:13:04.839393 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.839079 2577 scope.go:117] "RemoveContainer" containerID="30423936cbebc52ff506e2e30876456e309f01a25cdde7c35a87d12af163210e" Sep 06 14:13:04.840705 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.840681 2577 generic.go:356] "Generic (PLEG): container finished" podID="27d64c34-6f87-45cc-89b3-665590b87530" containerID="ce53105e917540d05c6498ff2921e9df1bda0d817f5ac48a282bab2e227669e7" exitCode=0 Sep 06 14:13:04.840795 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.840715 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" Sep 06 14:13:04.840795 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.840754 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" event={"ID":"27d64c34-6f87-45cc-89b3-665590b87530","Type":"ContainerDied","Data":"ce53105e917540d05c6498ff2921e9df1bda0d817f5ac48a282bab2e227669e7"} Sep 06 14:13:04.840875 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.840812 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll" event={"ID":"27d64c34-6f87-45cc-89b3-665590b87530","Type":"ContainerDied","Data":"4ea0a04d6516485e63336175a063f8fddda81d8591da1e827b54d6e2eb4f55d4"} Sep 06 14:13:04.841033 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.841008 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" podUID="7bab5853-37f4-443f-aaf8-3dc5356c1eb4" containerName="istio-proxy" containerID="cri-o://efefe9e9f8aaf98186a3841bfc5843927dba1895eb321586b4a04ace47163c36" gracePeriod=30 Sep 06 14:13:04.849456 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.849435 2577 scope.go:117] "RemoveContainer" containerID="30423936cbebc52ff506e2e30876456e309f01a25cdde7c35a87d12af163210e" Sep 06 14:13:04.849718 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:13:04.849698 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30423936cbebc52ff506e2e30876456e309f01a25cdde7c35a87d12af163210e\": container with ID starting with 30423936cbebc52ff506e2e30876456e309f01a25cdde7c35a87d12af163210e not found: ID does not exist" containerID="30423936cbebc52ff506e2e30876456e309f01a25cdde7c35a87d12af163210e" Sep 06 14:13:04.849822 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.849726 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30423936cbebc52ff506e2e30876456e309f01a25cdde7c35a87d12af163210e"} err="failed to get container status \"30423936cbebc52ff506e2e30876456e309f01a25cdde7c35a87d12af163210e\": rpc error: code = NotFound desc = could not find container \"30423936cbebc52ff506e2e30876456e309f01a25cdde7c35a87d12af163210e\": container with ID starting with 30423936cbebc52ff506e2e30876456e309f01a25cdde7c35a87d12af163210e not found: ID does not exist" Sep 06 14:13:04.849822 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.849743 2577 scope.go:117] "RemoveContainer" containerID="ce53105e917540d05c6498ff2921e9df1bda0d817f5ac48a282bab2e227669e7" Sep 06 14:13:04.857726 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.857698 2577 scope.go:117] "RemoveContainer" containerID="ce53105e917540d05c6498ff2921e9df1bda0d817f5ac48a282bab2e227669e7" Sep 06 14:13:04.857995 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:13:04.857976 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce53105e917540d05c6498ff2921e9df1bda0d817f5ac48a282bab2e227669e7\": container with ID starting with ce53105e917540d05c6498ff2921e9df1bda0d817f5ac48a282bab2e227669e7 not found: ID does not exist" containerID="ce53105e917540d05c6498ff2921e9df1bda0d817f5ac48a282bab2e227669e7" Sep 06 14:13:04.858083 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.858003 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce53105e917540d05c6498ff2921e9df1bda0d817f5ac48a282bab2e227669e7"} err="failed to get container status \"ce53105e917540d05c6498ff2921e9df1bda0d817f5ac48a282bab2e227669e7\": rpc error: code = NotFound desc = could not find container \"ce53105e917540d05c6498ff2921e9df1bda0d817f5ac48a282bab2e227669e7\": container with ID starting with ce53105e917540d05c6498ff2921e9df1bda0d817f5ac48a282bab2e227669e7 not found: ID does not exist" Sep 06 14:13:04.877856 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.877832 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr"] Sep 06 14:13:04.881593 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.881571 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b1ed35f1-56f8886d85-74bgr"] Sep 06 14:13:04.890715 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.890693 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll"] Sep 06 14:13:04.894279 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:04.894258 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b1ed35f1-7cb8c5b58b-tz2ll"] Sep 06 14:13:05.002092 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:05.002061 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:05.073728 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:05.073640 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27d64c34-6f87-45cc-89b3-665590b87530" path="/var/lib/kubelet/pods/27d64c34-6f87-45cc-89b3-665590b87530/volumes" Sep 06 14:13:05.074123 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:05.074104 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bdf53fd-6ef7-4494-b778-546b7372e8a3" path="/var/lib/kubelet/pods/2bdf53fd-6ef7-4494-b778-546b7372e8a3/volumes" Sep 06 14:13:06.191161 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:06.191127 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:10.100234 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.100210 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:13:10.168045 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.167970 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-workload-certs\") pod \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " Sep 06 14:13:10.168045 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.168028 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-podinfo\") pod \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " Sep 06 14:13:10.168239 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.168085 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-data\") pod \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " Sep 06 14:13:10.168239 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.168128 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-workload-socket\") pod \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " Sep 06 14:13:10.168239 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.168152 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-token\") pod \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " Sep 06 14:13:10.168239 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.168180 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istiod-ca-cert\") pod \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " Sep 06 14:13:10.168239 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.168196 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7bab5853-37f4-443f-aaf8-3dc5356c1eb4" (UID: "7bab5853-37f4-443f-aaf8-3dc5356c1eb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:13:10.168495 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.168369 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-credential-socket\") pod \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " Sep 06 14:13:10.168495 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.168455 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-data" (OuterVolumeSpecName: "istio-data") pod "7bab5853-37f4-443f-aaf8-3dc5356c1eb4" (UID: "7bab5853-37f4-443f-aaf8-3dc5356c1eb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:13:10.168495 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.168463 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-envoy\") pod \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " Sep 06 14:13:10.168495 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.168469 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7bab5853-37f4-443f-aaf8-3dc5356c1eb4" (UID: "7bab5853-37f4-443f-aaf8-3dc5356c1eb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:13:10.168691 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.168506 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9wrs\" (UniqueName: \"kubernetes.io/projected/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-kube-api-access-n9wrs\") pod \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\" (UID: \"7bab5853-37f4-443f-aaf8-3dc5356c1eb4\") " Sep 06 14:13:10.168691 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.168546 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7bab5853-37f4-443f-aaf8-3dc5356c1eb4" (UID: "7bab5853-37f4-443f-aaf8-3dc5356c1eb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:13:10.168789 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.168690 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7bab5853-37f4-443f-aaf8-3dc5356c1eb4" (UID: "7bab5853-37f4-443f-aaf8-3dc5356c1eb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:13:10.169108 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.169064 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-workload-certs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:13:10.169108 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.169089 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-data\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:13:10.169108 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.169102 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-workload-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:13:10.169334 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.169115 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istiod-ca-cert\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:13:10.169334 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.169128 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-credential-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:13:10.170332 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.170305 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7bab5853-37f4-443f-aaf8-3dc5356c1eb4" (UID: "7bab5853-37f4-443f-aaf8-3dc5356c1eb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 14:13:10.170532 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.170508 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7bab5853-37f4-443f-aaf8-3dc5356c1eb4" (UID: "7bab5853-37f4-443f-aaf8-3dc5356c1eb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:13:10.170718 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.170688 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-token" (OuterVolumeSpecName: "istio-token") pod "7bab5853-37f4-443f-aaf8-3dc5356c1eb4" (UID: "7bab5853-37f4-443f-aaf8-3dc5356c1eb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:13:10.170842 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.170821 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-kube-api-access-n9wrs" (OuterVolumeSpecName: "kube-api-access-n9wrs") pod "7bab5853-37f4-443f-aaf8-3dc5356c1eb4" (UID: "7bab5853-37f4-443f-aaf8-3dc5356c1eb4"). InnerVolumeSpecName "kube-api-access-n9wrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:13:10.269721 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.269687 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-envoy\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:13:10.269721 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.269715 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9wrs\" (UniqueName: \"kubernetes.io/projected/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-kube-api-access-n9wrs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:13:10.269721 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.269725 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-podinfo\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:13:10.269982 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.269733 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bab5853-37f4-443f-aaf8-3dc5356c1eb4-istio-token\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:13:10.593745 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.593713 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:13:10.711209 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.711172 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:10.874600 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.874505 2577 generic.go:356] "Generic (PLEG): container finished" podID="7bab5853-37f4-443f-aaf8-3dc5356c1eb4" containerID="efefe9e9f8aaf98186a3841bfc5843927dba1895eb321586b4a04ace47163c36" exitCode=0 Sep 06 14:13:10.874600 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.874555 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" Sep 06 14:13:10.874600 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.874590 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" event={"ID":"7bab5853-37f4-443f-aaf8-3dc5356c1eb4","Type":"ContainerDied","Data":"efefe9e9f8aaf98186a3841bfc5843927dba1895eb321586b4a04ace47163c36"} Sep 06 14:13:10.874870 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.874651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl" event={"ID":"7bab5853-37f4-443f-aaf8-3dc5356c1eb4","Type":"ContainerDied","Data":"c38e78856c97c2b81d7a868744745d1d2312bc82a3f36e6b68edb8715fced1c6"} Sep 06 14:13:10.874870 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.874674 2577 scope.go:117] "RemoveContainer" containerID="efefe9e9f8aaf98186a3841bfc5843927dba1895eb321586b4a04ace47163c36" Sep 06 14:13:10.883865 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.883840 2577 scope.go:117] "RemoveContainer" containerID="9efcb5e249566afcf62b116c2dfdb10469a098ead3305cb04028ab50a16a0e50" Sep 06 14:13:10.891815 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.891800 2577 scope.go:117] "RemoveContainer" containerID="efefe9e9f8aaf98186a3841bfc5843927dba1895eb321586b4a04ace47163c36" Sep 06 14:13:10.892371 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:13:10.892329 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efefe9e9f8aaf98186a3841bfc5843927dba1895eb321586b4a04ace47163c36\": container with ID starting with efefe9e9f8aaf98186a3841bfc5843927dba1895eb321586b4a04ace47163c36 not found: ID does not exist" containerID="efefe9e9f8aaf98186a3841bfc5843927dba1895eb321586b4a04ace47163c36" Sep 06 14:13:10.892371 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.892363 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efefe9e9f8aaf98186a3841bfc5843927dba1895eb321586b4a04ace47163c36"} err="failed to get container status \"efefe9e9f8aaf98186a3841bfc5843927dba1895eb321586b4a04ace47163c36\": rpc error: code = NotFound desc = could not find container \"efefe9e9f8aaf98186a3841bfc5843927dba1895eb321586b4a04ace47163c36\": container with ID starting with efefe9e9f8aaf98186a3841bfc5843927dba1895eb321586b4a04ace47163c36 not found: ID does not exist" Sep 06 14:13:10.892371 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.892386 2577 scope.go:117] "RemoveContainer" containerID="9efcb5e249566afcf62b116c2dfdb10469a098ead3305cb04028ab50a16a0e50" Sep 06 14:13:10.892810 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:13:10.892772 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9efcb5e249566afcf62b116c2dfdb10469a098ead3305cb04028ab50a16a0e50\": container with ID starting with 9efcb5e249566afcf62b116c2dfdb10469a098ead3305cb04028ab50a16a0e50 not found: ID does not exist" containerID="9efcb5e249566afcf62b116c2dfdb10469a098ead3305cb04028ab50a16a0e50" Sep 06 14:13:10.892952 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.892816 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9efcb5e249566afcf62b116c2dfdb10469a098ead3305cb04028ab50a16a0e50"} err="failed to get container status \"9efcb5e249566afcf62b116c2dfdb10469a098ead3305cb04028ab50a16a0e50\": rpc error: code = NotFound desc = could not find container \"9efcb5e249566afcf62b116c2dfdb10469a098ead3305cb04028ab50a16a0e50\": container with ID starting with 9efcb5e249566afcf62b116c2dfdb10469a098ead3305cb04028ab50a16a0e50 not found: ID does not exist" Sep 06 14:13:10.895335 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.895314 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl"] Sep 06 14:13:10.900062 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:10.900042 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b1ed35f1-openshift-default-f9489d756-cwnpl"] Sep 06 14:13:11.075573 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:11.075536 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bab5853-37f4-443f-aaf8-3dc5356c1eb4" path="/var/lib/kubelet/pods/7bab5853-37f4-443f-aaf8-3dc5356c1eb4/volumes" Sep 06 14:13:12.026202 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:12.026164 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:12.123812 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:12.123778 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:12.224978 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:12.224943 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:12.324116 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:12.324039 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:12.658201 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:12.658116 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:23.089962 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:23.089931 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:13:24.177803 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:24.177770 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:24.412996 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:24.412957 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:24.840300 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:24.840264 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:25.095575 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.095493 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq"] Sep 06 14:13:25.096177 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096152 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b169e109-e833-4fff-a5a8-97c1d8125c9f" containerName="revoke-keys" Sep 06 14:13:25.096177 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096176 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b169e109-e833-4fff-a5a8-97c1d8125c9f" containerName="revoke-keys" Sep 06 14:13:25.096325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096200 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9" containerName="payload-pre-processing" Sep 06 14:13:25.096325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096209 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9" containerName="payload-pre-processing" Sep 06 14:13:25.096325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096220 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d64c34-6f87-45cc-89b3-665590b87530" containerName="payload-pre-processing" Sep 06 14:13:25.096325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096229 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d64c34-6f87-45cc-89b3-665590b87530" containerName="payload-pre-processing" Sep 06 14:13:25.096325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096261 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bab5853-37f4-443f-aaf8-3dc5356c1eb4" containerName="istio-proxy" Sep 06 14:13:25.096325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096269 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bab5853-37f4-443f-aaf8-3dc5356c1eb4" containerName="istio-proxy" Sep 06 14:13:25.096325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096280 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bdf53fd-6ef7-4494-b778-546b7372e8a3" containerName="payload-processing" Sep 06 14:13:25.096325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096288 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bdf53fd-6ef7-4494-b778-546b7372e8a3" containerName="payload-processing" Sep 06 14:13:25.096325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096297 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bab5853-37f4-443f-aaf8-3dc5356c1eb4" containerName="istio-proxy" Sep 06 14:13:25.096325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096305 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bab5853-37f4-443f-aaf8-3dc5356c1eb4" containerName="istio-proxy" Sep 06 14:13:25.096633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096329 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64788558-c059-4cda-a753-64d3393bfd88" containerName="payload-processing" Sep 06 14:13:25.096633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096338 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="64788558-c059-4cda-a753-64d3393bfd88" containerName="payload-processing" Sep 06 14:13:25.096633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096427 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="017b9ae0-0d9e-42cf-88f3-28aa6ab81ab9" containerName="payload-pre-processing" Sep 06 14:13:25.096633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096440 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b169e109-e833-4fff-a5a8-97c1d8125c9f" containerName="revoke-keys" Sep 06 14:13:25.096633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096448 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="64788558-c059-4cda-a753-64d3393bfd88" containerName="payload-processing" Sep 06 14:13:25.096633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096457 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bab5853-37f4-443f-aaf8-3dc5356c1eb4" containerName="istio-proxy" Sep 06 14:13:25.096633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096465 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bdf53fd-6ef7-4494-b778-546b7372e8a3" containerName="payload-processing" Sep 06 14:13:25.096633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096472 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d64c34-6f87-45cc-89b3-665590b87530" containerName="payload-pre-processing" Sep 06 14:13:25.096633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.096481 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bab5853-37f4-443f-aaf8-3dc5356c1eb4" containerName="istio-proxy" Sep 06 14:13:25.099615 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.099596 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.103021 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.103004 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-dfca8b-openshift-default-dockercfg-zmpjg\"" Sep 06 14:13:25.122532 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.122502 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq"] Sep 06 14:13:25.139910 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.139886 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:25.207724 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.207690 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-workload-certs\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.208207 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.207786 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r442\" (UniqueName: \"kubernetes.io/projected/d13af05b-d79f-4675-b973-1ac87630cc57-kube-api-access-9r442\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.208207 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.207973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-credential-socket\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.208207 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.208031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13af05b-d79f-4675-b973-1ac87630cc57-istio-token\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.208207 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.208068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13af05b-d79f-4675-b973-1ac87630cc57-istiod-ca-cert\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.208207 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.208179 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-istio-data\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.208485 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.208242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13af05b-d79f-4675-b973-1ac87630cc57-istio-podinfo\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.208485 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.208316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-workload-socket\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.208485 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.208355 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-istio-envoy\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.309455 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.309423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-credential-socket\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.309455 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.309455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13af05b-d79f-4675-b973-1ac87630cc57-istio-token\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.309680 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.309471 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13af05b-d79f-4675-b973-1ac87630cc57-istiod-ca-cert\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.309680 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.309497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-istio-data\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.309680 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.309523 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13af05b-d79f-4675-b973-1ac87630cc57-istio-podinfo\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.309680 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.309551 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-workload-socket\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.309680 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.309568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-istio-envoy\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.309680 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.309599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-workload-certs\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.309680 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.309656 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r442\" (UniqueName: \"kubernetes.io/projected/d13af05b-d79f-4675-b973-1ac87630cc57-kube-api-access-9r442\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.310107 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.310079 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-istio-data\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.310173 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.310127 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-credential-socket\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.310235 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.310173 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-workload-socket\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.310291 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.310275 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13af05b-d79f-4675-b973-1ac87630cc57-istiod-ca-cert\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.310336 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.310304 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-workload-certs\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.311947 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.311929 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-istio-envoy\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.312249 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.312232 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13af05b-d79f-4675-b973-1ac87630cc57-istio-podinfo\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.316722 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.316693 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r442\" (UniqueName: \"kubernetes.io/projected/d13af05b-d79f-4675-b973-1ac87630cc57-kube-api-access-9r442\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.316962 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.316943 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13af05b-d79f-4675-b973-1ac87630cc57-istio-token\") pod \"e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.323438 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.323418 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:25.410221 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.410134 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:25.684027 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.684004 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq"] Sep 06 14:13:25.685466 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:13:25.685443 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13af05b_d79f_4675_b973_1ac87630cc57.slice/crio-79c23a6c7c2ad6435c5c0daf25bab59ae69c30a67351b55a8ce327cef481c1f4 WatchSource:0}: Error finding container 79c23a6c7c2ad6435c5c0daf25bab59ae69c30a67351b55a8ce327cef481c1f4: Status 404 returned error can't find the container with id 79c23a6c7c2ad6435c5c0daf25bab59ae69c30a67351b55a8ce327cef481c1f4 Sep 06 14:13:25.687591 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.687557 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:13:25.687681 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.687635 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:13:25.687744 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.687682 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:13:25.952733 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.952649 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" event={"ID":"d13af05b-d79f-4675-b973-1ac87630cc57","Type":"ContainerStarted","Data":"76e6e77e7768b811258d9815d9a2b4e280d367bba8931c5d01a5e45a7a346bbd"} Sep 06 14:13:25.952733 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:25.952717 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" event={"ID":"d13af05b-d79f-4675-b973-1ac87630cc57","Type":"ContainerStarted","Data":"79c23a6c7c2ad6435c5c0daf25bab59ae69c30a67351b55a8ce327cef481c1f4"} Sep 06 14:13:26.411390 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:26.411354 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:26.416300 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:26.416274 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:26.434697 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:26.434652 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" podStartSLOduration=1.434636774 podStartE2EDuration="1.434636774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:13:25.969379738 +0000 UTC m=+937.424868600" watchObservedRunningTime="2026-09-06 14:13:26.434636774 +0000 UTC m=+937.890125692" Sep 06 14:13:26.957399 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:26.957366 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:26.958198 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:26.958182 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:27.804372 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:27.804338 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:27.905468 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:27.905434 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:28.950102 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:28.950072 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:29.051040 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.051007 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:29.150212 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.150178 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:29.520631 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.520601 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:29.612376 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.612340 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:29.619685 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.619658 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7"] Sep 06 14:13:29.623340 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.623325 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" Sep 06 14:13:29.625731 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.625714 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-dfca8b-dockercfg-n8jhl\"" Sep 06 14:13:29.626571 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.626552 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-dfca8b\"" Sep 06 14:13:29.632569 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.632548 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7"] Sep 06 14:13:29.651014 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.650991 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq"] Sep 06 14:13:29.654891 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.654866 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" Sep 06 14:13:29.663732 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.663710 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq"] Sep 06 14:13:29.751299 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.751259 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ab635f-56a7-45d9-b841-c7ba5d466276-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7\" (UID: \"82ab635f-56a7-45d9-b841-c7ba5d466276\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" Sep 06 14:13:29.751299 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.751302 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gn4s\" (UniqueName: \"kubernetes.io/projected/833f5811-a842-4f3b-85b0-9522aa60bfd7-kube-api-access-9gn4s\") pod \"payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq\" (UID: \"833f5811-a842-4f3b-85b0-9522aa60bfd7\") " pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" Sep 06 14:13:29.751514 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.751376 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf6nm\" (UniqueName: \"kubernetes.io/projected/82ab635f-56a7-45d9-b841-c7ba5d466276-kube-api-access-hf6nm\") pod \"payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7\" (UID: \"82ab635f-56a7-45d9-b841-c7ba5d466276\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" Sep 06 14:13:29.751514 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.751403 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833f5811-a842-4f3b-85b0-9522aa60bfd7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq\" (UID: \"833f5811-a842-4f3b-85b0-9522aa60bfd7\") " pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" Sep 06 14:13:29.852635 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.852543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ab635f-56a7-45d9-b841-c7ba5d466276-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7\" (UID: \"82ab635f-56a7-45d9-b841-c7ba5d466276\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" Sep 06 14:13:29.852635 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.852599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gn4s\" (UniqueName: \"kubernetes.io/projected/833f5811-a842-4f3b-85b0-9522aa60bfd7-kube-api-access-9gn4s\") pod \"payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq\" (UID: \"833f5811-a842-4f3b-85b0-9522aa60bfd7\") " pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" Sep 06 14:13:29.852859 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.852639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf6nm\" (UniqueName: \"kubernetes.io/projected/82ab635f-56a7-45d9-b841-c7ba5d466276-kube-api-access-hf6nm\") pod \"payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7\" (UID: \"82ab635f-56a7-45d9-b841-c7ba5d466276\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" Sep 06 14:13:29.852859 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.852663 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833f5811-a842-4f3b-85b0-9522aa60bfd7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq\" (UID: \"833f5811-a842-4f3b-85b0-9522aa60bfd7\") " pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" Sep 06 14:13:29.853309 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.853286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ab635f-56a7-45d9-b841-c7ba5d466276-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7\" (UID: \"82ab635f-56a7-45d9-b841-c7ba5d466276\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" Sep 06 14:13:29.853440 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.853419 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833f5811-a842-4f3b-85b0-9522aa60bfd7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq\" (UID: \"833f5811-a842-4f3b-85b0-9522aa60bfd7\") " pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" Sep 06 14:13:29.860083 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.860057 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gn4s\" (UniqueName: \"kubernetes.io/projected/833f5811-a842-4f3b-85b0-9522aa60bfd7-kube-api-access-9gn4s\") pod \"payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq\" (UID: \"833f5811-a842-4f3b-85b0-9522aa60bfd7\") " pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" Sep 06 14:13:29.860179 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.860138 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf6nm\" (UniqueName: \"kubernetes.io/projected/82ab635f-56a7-45d9-b841-c7ba5d466276-kube-api-access-hf6nm\") pod \"payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7\" (UID: \"82ab635f-56a7-45d9-b841-c7ba5d466276\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" Sep 06 14:13:29.937227 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.937190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" Sep 06 14:13:29.967489 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:29.967450 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" Sep 06 14:13:30.210212 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:30.210185 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7"] Sep 06 14:13:30.211171 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:13:30.211145 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ab635f_56a7_45d9_b841_c7ba5d466276.slice/crio-fded8ee7ecc8307371ea0d18add6f912fa6ea07fe2a84ff9e548cc702f4e6e7f WatchSource:0}: Error finding container fded8ee7ecc8307371ea0d18add6f912fa6ea07fe2a84ff9e548cc702f4e6e7f: Status 404 returned error can't find the container with id fded8ee7ecc8307371ea0d18add6f912fa6ea07fe2a84ff9e548cc702f4e6e7f Sep 06 14:13:30.273240 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:30.273216 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq"] Sep 06 14:13:30.274744 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:13:30.274721 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833f5811_a842_4f3b_85b0_9522aa60bfd7.slice/crio-df413a4513814433067ede3372bbfa244085f5cb0dc232ad27f232d3a2088244 WatchSource:0}: Error finding container df413a4513814433067ede3372bbfa244085f5cb0dc232ad27f232d3a2088244: Status 404 returned error can't find the container with id df413a4513814433067ede3372bbfa244085f5cb0dc232ad27f232d3a2088244 Sep 06 14:13:30.978867 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:30.978824 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" event={"ID":"833f5811-a842-4f3b-85b0-9522aa60bfd7","Type":"ContainerStarted","Data":"35a63226c67844e102c5478d75e7721908f314ecbaeaa1cd933237580f1b08bb"} Sep 06 14:13:30.979323 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:30.978930 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" event={"ID":"833f5811-a842-4f3b-85b0-9522aa60bfd7","Type":"ContainerStarted","Data":"df413a4513814433067ede3372bbfa244085f5cb0dc232ad27f232d3a2088244"} Sep 06 14:13:30.979323 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:30.978960 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" Sep 06 14:13:30.980484 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:30.980459 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" event={"ID":"82ab635f-56a7-45d9-b841-c7ba5d466276","Type":"ContainerStarted","Data":"158bddd5b96199516c837b4bb5f933635eeddf5569c0bea1ca3bcf2dd73bf7c7"} Sep 06 14:13:30.980615 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:30.980499 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" event={"ID":"82ab635f-56a7-45d9-b841-c7ba5d466276","Type":"ContainerStarted","Data":"fded8ee7ecc8307371ea0d18add6f912fa6ea07fe2a84ff9e548cc702f4e6e7f"} Sep 06 14:13:30.980615 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:30.980603 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" Sep 06 14:13:30.995269 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:30.995219 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" podStartSLOduration=1.995203915 podStartE2EDuration="1.995203915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:13:30.993249263 +0000 UTC m=+942.448738138" watchObservedRunningTime="2026-09-06 14:13:30.995203915 +0000 UTC m=+942.450692778" Sep 06 14:13:31.008389 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:31.008344 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" podStartSLOduration=2.008330659 podStartE2EDuration="2.008330659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:13:31.006169219 +0000 UTC m=+942.461658081" watchObservedRunningTime="2026-09-06 14:13:31.008330659 +0000 UTC m=+942.463819521" Sep 06 14:13:37.498415 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:37.498376 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:13:38.529451 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:38.529419 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:41.988047 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:41.988019 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" Sep 06 14:13:41.988416 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:41.988084 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" Sep 06 14:13:44.198444 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:44.198373 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:44.254703 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:44.254654 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:44.332389 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:44.332362 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:44.429693 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:44.429656 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:48.096129 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:48.096087 2577 generic.go:356] "Generic (PLEG): container finished" podID="d13af05b-d79f-4675-b973-1ac87630cc57" containerID="76e6e77e7768b811258d9815d9a2b4e280d367bba8931c5d01a5e45a7a346bbd" exitCode=0 Sep 06 14:13:48.096525 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:48.096176 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" event={"ID":"d13af05b-d79f-4675-b973-1ac87630cc57","Type":"ContainerDied","Data":"76e6e77e7768b811258d9815d9a2b4e280d367bba8931c5d01a5e45a7a346bbd"} Sep 06 14:13:48.096591 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:48.096551 2577 scope.go:117] "RemoveContainer" containerID="76e6e77e7768b811258d9815d9a2b4e280d367bba8931c5d01a5e45a7a346bbd" Sep 06 14:13:48.097629 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:48.097603 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:13:48.097697 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:48.097665 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:13:48.097743 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:48.097698 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:13:49.103566 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:49.103536 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" event={"ID":"d13af05b-d79f-4675-b973-1ac87630cc57","Type":"ContainerStarted","Data":"035adb5478d6358b67e595dc412ab944fb327b1a37c78e64bc3a507146e1d5f1"} Sep 06 14:13:49.410616 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:49.410531 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:49.410839 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:49.410813 2577 patch_prober.go:28] interesting pod/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 06 14:13:49.410887 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:49.410873 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" podUID="d13af05b-d79f-4675-b973-1ac87630cc57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 06 14:13:50.410726 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:50.410693 2577 patch_prober.go:28] interesting pod/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 06 14:13:50.411144 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:50.410752 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" podUID="d13af05b-d79f-4675-b973-1ac87630cc57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 06 14:13:51.410794 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:51.410761 2577 patch_prober.go:28] interesting pod/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 06 14:13:51.411172 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:51.410825 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" podUID="d13af05b-d79f-4675-b973-1ac87630cc57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 06 14:13:52.411083 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:52.411048 2577 patch_prober.go:28] interesting pod/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 06 14:13:52.411451 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:52.411134 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" podUID="d13af05b-d79f-4675-b973-1ac87630cc57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 06 14:13:52.586473 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:52.586437 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:13:52.722577 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:52.722509 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:13:53.411261 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:53.411229 2577 patch_prober.go:28] interesting pod/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 06 14:13:53.411648 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:53.411292 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" podUID="d13af05b-d79f-4675-b973-1ac87630cc57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 06 14:13:54.627415 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:54.627382 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:54.627888 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:54.627850 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:54.629003 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:54.628953 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:13:55.211543 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:13:55.211512 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:00.167589 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:00.167556 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:00.227690 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:00.227619 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:06.004933 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:06.004866 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:06.110084 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:06.110048 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:06.193340 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:06.193284 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:06.247879 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:06.247851 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:09.286857 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:09.286829 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:14:16.659837 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:16.659752 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:16.732037 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:16.732000 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq"] Sep 06 14:14:16.732317 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:16.732289 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" podUID="833f5811-a842-4f3b-85b0-9522aa60bfd7" containerName="payload-processing" containerID="cri-o://35a63226c67844e102c5478d75e7721908f314ecbaeaa1cd933237580f1b08bb" gracePeriod=30 Sep 06 14:14:16.737963 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:16.737935 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7"] Sep 06 14:14:16.738195 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:16.738154 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" podUID="82ab635f-56a7-45d9-b841-c7ba5d466276" containerName="payload-pre-processing" containerID="cri-o://158bddd5b96199516c837b4bb5f933635eeddf5569c0bea1ca3bcf2dd73bf7c7" gracePeriod=30 Sep 06 14:14:16.738818 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:16.738511 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:16.801841 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:16.801740 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:16.881201 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:16.881168 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:16.940445 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:16.940400 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:16.997008 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:16.996973 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" Sep 06 14:14:17.025685 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.025659 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" Sep 06 14:14:17.119002 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.118967 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hf6nm\" (UniqueName: \"kubernetes.io/projected/82ab635f-56a7-45d9-b841-c7ba5d466276-kube-api-access-hf6nm\") pod \"82ab635f-56a7-45d9-b841-c7ba5d466276\" (UID: \"82ab635f-56a7-45d9-b841-c7ba5d466276\") " Sep 06 14:14:17.119180 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.119031 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833f5811-a842-4f3b-85b0-9522aa60bfd7-plugins-config-volume\") pod \"833f5811-a842-4f3b-85b0-9522aa60bfd7\" (UID: \"833f5811-a842-4f3b-85b0-9522aa60bfd7\") " Sep 06 14:14:17.119180 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.119061 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ab635f-56a7-45d9-b841-c7ba5d466276-plugins-config-volume\") pod \"82ab635f-56a7-45d9-b841-c7ba5d466276\" (UID: \"82ab635f-56a7-45d9-b841-c7ba5d466276\") " Sep 06 14:14:17.119180 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.119109 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gn4s\" (UniqueName: \"kubernetes.io/projected/833f5811-a842-4f3b-85b0-9522aa60bfd7-kube-api-access-9gn4s\") pod \"833f5811-a842-4f3b-85b0-9522aa60bfd7\" (UID: \"833f5811-a842-4f3b-85b0-9522aa60bfd7\") " Sep 06 14:14:17.119427 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.119402 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833f5811-a842-4f3b-85b0-9522aa60bfd7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "833f5811-a842-4f3b-85b0-9522aa60bfd7" (UID: "833f5811-a842-4f3b-85b0-9522aa60bfd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:14:17.119538 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.119456 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ab635f-56a7-45d9-b841-c7ba5d466276-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82ab635f-56a7-45d9-b841-c7ba5d466276" (UID: "82ab635f-56a7-45d9-b841-c7ba5d466276"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:14:17.121057 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.121036 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ab635f-56a7-45d9-b841-c7ba5d466276-kube-api-access-hf6nm" (OuterVolumeSpecName: "kube-api-access-hf6nm") pod "82ab635f-56a7-45d9-b841-c7ba5d466276" (UID: "82ab635f-56a7-45d9-b841-c7ba5d466276"). InnerVolumeSpecName "kube-api-access-hf6nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:14:17.121130 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.121094 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833f5811-a842-4f3b-85b0-9522aa60bfd7-kube-api-access-9gn4s" (OuterVolumeSpecName: "kube-api-access-9gn4s") pod "833f5811-a842-4f3b-85b0-9522aa60bfd7" (UID: "833f5811-a842-4f3b-85b0-9522aa60bfd7"). InnerVolumeSpecName "kube-api-access-9gn4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:14:17.220889 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.220815 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gn4s\" (UniqueName: \"kubernetes.io/projected/833f5811-a842-4f3b-85b0-9522aa60bfd7-kube-api-access-9gn4s\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:17.220889 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.220843 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf6nm\" (UniqueName: \"kubernetes.io/projected/82ab635f-56a7-45d9-b841-c7ba5d466276-kube-api-access-hf6nm\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:17.220889 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.220853 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833f5811-a842-4f3b-85b0-9522aa60bfd7-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:17.220889 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.220861 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ab635f-56a7-45d9-b841-c7ba5d466276-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:17.229886 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.229858 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:17.250871 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.250836 2577 generic.go:356] "Generic (PLEG): container finished" podID="82ab635f-56a7-45d9-b841-c7ba5d466276" containerID="158bddd5b96199516c837b4bb5f933635eeddf5569c0bea1ca3bcf2dd73bf7c7" exitCode=0 Sep 06 14:14:17.251025 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.250880 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" Sep 06 14:14:17.251025 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.250894 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" event={"ID":"82ab635f-56a7-45d9-b841-c7ba5d466276","Type":"ContainerDied","Data":"158bddd5b96199516c837b4bb5f933635eeddf5569c0bea1ca3bcf2dd73bf7c7"} Sep 06 14:14:17.251025 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.250976 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7" event={"ID":"82ab635f-56a7-45d9-b841-c7ba5d466276","Type":"ContainerDied","Data":"fded8ee7ecc8307371ea0d18add6f912fa6ea07fe2a84ff9e548cc702f4e6e7f"} Sep 06 14:14:17.251025 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.251007 2577 scope.go:117] "RemoveContainer" containerID="158bddd5b96199516c837b4bb5f933635eeddf5569c0bea1ca3bcf2dd73bf7c7" Sep 06 14:14:17.252410 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.252384 2577 generic.go:356] "Generic (PLEG): container finished" podID="833f5811-a842-4f3b-85b0-9522aa60bfd7" containerID="35a63226c67844e102c5478d75e7721908f314ecbaeaa1cd933237580f1b08bb" exitCode=0 Sep 06 14:14:17.252525 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.252440 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" Sep 06 14:14:17.252525 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.252461 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" event={"ID":"833f5811-a842-4f3b-85b0-9522aa60bfd7","Type":"ContainerDied","Data":"35a63226c67844e102c5478d75e7721908f314ecbaeaa1cd933237580f1b08bb"} Sep 06 14:14:17.252525 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.252498 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq" event={"ID":"833f5811-a842-4f3b-85b0-9522aa60bfd7","Type":"ContainerDied","Data":"df413a4513814433067ede3372bbfa244085f5cb0dc232ad27f232d3a2088244"} Sep 06 14:14:17.262137 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.262121 2577 scope.go:117] "RemoveContainer" containerID="158bddd5b96199516c837b4bb5f933635eeddf5569c0bea1ca3bcf2dd73bf7c7" Sep 06 14:14:17.262476 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:14:17.262415 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158bddd5b96199516c837b4bb5f933635eeddf5569c0bea1ca3bcf2dd73bf7c7\": container with ID starting with 158bddd5b96199516c837b4bb5f933635eeddf5569c0bea1ca3bcf2dd73bf7c7 not found: ID does not exist" containerID="158bddd5b96199516c837b4bb5f933635eeddf5569c0bea1ca3bcf2dd73bf7c7" Sep 06 14:14:17.262476 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.262450 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158bddd5b96199516c837b4bb5f933635eeddf5569c0bea1ca3bcf2dd73bf7c7"} err="failed to get container status \"158bddd5b96199516c837b4bb5f933635eeddf5569c0bea1ca3bcf2dd73bf7c7\": rpc error: code = NotFound desc = could not find container \"158bddd5b96199516c837b4bb5f933635eeddf5569c0bea1ca3bcf2dd73bf7c7\": container with ID starting with 158bddd5b96199516c837b4bb5f933635eeddf5569c0bea1ca3bcf2dd73bf7c7 not found: ID does not exist" Sep 06 14:14:17.262476 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.262475 2577 scope.go:117] "RemoveContainer" containerID="35a63226c67844e102c5478d75e7721908f314ecbaeaa1cd933237580f1b08bb" Sep 06 14:14:17.271987 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.271968 2577 scope.go:117] "RemoveContainer" containerID="35a63226c67844e102c5478d75e7721908f314ecbaeaa1cd933237580f1b08bb" Sep 06 14:14:17.272291 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:14:17.272274 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a63226c67844e102c5478d75e7721908f314ecbaeaa1cd933237580f1b08bb\": container with ID starting with 35a63226c67844e102c5478d75e7721908f314ecbaeaa1cd933237580f1b08bb not found: ID does not exist" containerID="35a63226c67844e102c5478d75e7721908f314ecbaeaa1cd933237580f1b08bb" Sep 06 14:14:17.272339 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.272299 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a63226c67844e102c5478d75e7721908f314ecbaeaa1cd933237580f1b08bb"} err="failed to get container status \"35a63226c67844e102c5478d75e7721908f314ecbaeaa1cd933237580f1b08bb\": rpc error: code = NotFound desc = could not find container \"35a63226c67844e102c5478d75e7721908f314ecbaeaa1cd933237580f1b08bb\": container with ID starting with 35a63226c67844e102c5478d75e7721908f314ecbaeaa1cd933237580f1b08bb not found: ID does not exist" Sep 06 14:14:17.273242 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.273224 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7"] Sep 06 14:14:17.276505 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.276487 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-dfca8b-6f485d86-7hdk7"] Sep 06 14:14:17.285949 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.285928 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq"] Sep 06 14:14:17.287765 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.287746 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-dfca8b-59db7989fc-h5gsq"] Sep 06 14:14:17.412173 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:17.412131 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:18.814209 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:18.814178 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:19.068751 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:19.068674 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ab635f-56a7-45d9-b841-c7ba5d466276" path="/var/lib/kubelet/pods/82ab635f-56a7-45d9-b841-c7ba5d466276/volumes" Sep 06 14:14:19.069069 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:19.069055 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="833f5811-a842-4f3b-85b0-9522aa60bfd7" path="/var/lib/kubelet/pods/833f5811-a842-4f3b-85b0-9522aa60bfd7/volumes" Sep 06 14:14:19.919223 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:19.919190 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:20.010241 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.010210 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw"] Sep 06 14:14:20.010676 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.010663 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ab635f-56a7-45d9-b841-c7ba5d466276" containerName="payload-pre-processing" Sep 06 14:14:20.010725 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.010677 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ab635f-56a7-45d9-b841-c7ba5d466276" containerName="payload-pre-processing" Sep 06 14:14:20.010725 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.010702 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833f5811-a842-4f3b-85b0-9522aa60bfd7" containerName="payload-processing" Sep 06 14:14:20.010725 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.010710 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="833f5811-a842-4f3b-85b0-9522aa60bfd7" containerName="payload-processing" Sep 06 14:14:20.010819 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.010767 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ab635f-56a7-45d9-b841-c7ba5d466276" containerName="payload-pre-processing" Sep 06 14:14:20.010819 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.010781 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="833f5811-a842-4f3b-85b0-9522aa60bfd7" containerName="payload-processing" Sep 06 14:14:20.014084 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.014065 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw" Sep 06 14:14:20.016316 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.016299 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\"" Sep 06 14:14:20.016369 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.016299 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 14:14:20.021470 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.021445 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw"] Sep 06 14:14:20.147578 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.147542 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae55c260-20c1-49be-a82b-c52a0f07a370-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw\" (UID: \"ae55c260-20c1-49be-a82b-c52a0f07a370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw" Sep 06 14:14:20.237582 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.237504 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:20.248201 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.248174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae55c260-20c1-49be-a82b-c52a0f07a370-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw\" (UID: \"ae55c260-20c1-49be-a82b-c52a0f07a370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw" Sep 06 14:14:20.248732 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.248711 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae55c260-20c1-49be-a82b-c52a0f07a370-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw\" (UID: \"ae55c260-20c1-49be-a82b-c52a0f07a370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw" Sep 06 14:14:20.326337 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.326308 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw" Sep 06 14:14:20.596015 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:20.595991 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw"] Sep 06 14:14:20.598129 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:14:20.598092 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae55c260_20c1_49be_a82b_c52a0f07a370.slice/crio-28f967f9791b3c7b51f1e66dcc628caff122eac7e2c13965b912873d7b667c00 WatchSource:0}: Error finding container 28f967f9791b3c7b51f1e66dcc628caff122eac7e2c13965b912873d7b667c00: Status 404 returned error can't find the container with id 28f967f9791b3c7b51f1e66dcc628caff122eac7e2c13965b912873d7b667c00 Sep 06 14:14:21.276189 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:21.276154 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw" event={"ID":"ae55c260-20c1-49be-a82b-c52a0f07a370","Type":"ContainerStarted","Data":"67a35d5a6e99cd8450a4e27664caec2c4843dcde10fd1a56736a5af749b8d56e"} Sep 06 14:14:21.276536 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:21.276232 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw" event={"ID":"ae55c260-20c1-49be-a82b-c52a0f07a370","Type":"ContainerStarted","Data":"28f967f9791b3c7b51f1e66dcc628caff122eac7e2c13965b912873d7b667c00"} Sep 06 14:14:21.290624 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:21.290569 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw" podStartSLOduration=2.290555208 podStartE2EDuration="2.290555208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:14:21.288944313 +0000 UTC m=+992.744433179" watchObservedRunningTime="2026-09-06 14:14:21.290555208 +0000 UTC m=+992.746044070" Sep 06 14:14:22.282707 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:22.282677 2577 generic.go:356] "Generic (PLEG): container finished" podID="ae55c260-20c1-49be-a82b-c52a0f07a370" containerID="67a35d5a6e99cd8450a4e27664caec2c4843dcde10fd1a56736a5af749b8d56e" exitCode=0 Sep 06 14:14:22.283104 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:22.282766 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw" event={"ID":"ae55c260-20c1-49be-a82b-c52a0f07a370","Type":"ContainerDied","Data":"67a35d5a6e99cd8450a4e27664caec2c4843dcde10fd1a56736a5af749b8d56e"} Sep 06 14:14:22.494621 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:22.494582 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:14:23.420433 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:23.420408 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw" Sep 06 14:14:23.477730 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:23.477705 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae55c260-20c1-49be-a82b-c52a0f07a370-maas-api-service-ca\") pod \"ae55c260-20c1-49be-a82b-c52a0f07a370\" (UID: \"ae55c260-20c1-49be-a82b-c52a0f07a370\") " Sep 06 14:14:23.478042 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:23.478020 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae55c260-20c1-49be-a82b-c52a0f07a370-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae55c260-20c1-49be-a82b-c52a0f07a370" (UID: "ae55c260-20c1-49be-a82b-c52a0f07a370"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:14:23.578958 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:23.578859 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae55c260-20c1-49be-a82b-c52a0f07a370-maas-api-service-ca\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:23.654956 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:23.654924 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:23.711730 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:23.711701 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:23.868862 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:23.868779 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:23.874960 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:23.874929 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq"] Sep 06 14:14:23.875175 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:23.875140 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" podUID="d13af05b-d79f-4675-b973-1ac87630cc57" containerName="istio-proxy" containerID="cri-o://035adb5478d6358b67e595dc412ab944fb327b1a37c78e64bc3a507146e1d5f1" gracePeriod=30 Sep 06 14:14:23.968749 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:23.968703 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:24.068746 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.068703 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:24.168317 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.168218 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:24.293836 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.293808 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw" Sep 06 14:14:24.294015 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.293837 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw" event={"ID":"ae55c260-20c1-49be-a82b-c52a0f07a370","Type":"ContainerDied","Data":"28f967f9791b3c7b51f1e66dcc628caff122eac7e2c13965b912873d7b667c00"} Sep 06 14:14:24.294015 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.293878 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28f967f9791b3c7b51f1e66dcc628caff122eac7e2c13965b912873d7b667c00" Sep 06 14:14:24.303085 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.303059 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:24.566192 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.566160 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j"] Sep 06 14:14:24.566854 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.566831 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae55c260-20c1-49be-a82b-c52a0f07a370" containerName="revoke-keys" Sep 06 14:14:24.566854 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.566855 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae55c260-20c1-49be-a82b-c52a0f07a370" containerName="revoke-keys" Sep 06 14:14:24.567078 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.566961 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae55c260-20c1-49be-a82b-c52a0f07a370" containerName="revoke-keys" Sep 06 14:14:24.569399 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.569378 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.571679 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.571660 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-4e801f-openshift-default-dockercfg-7mq79\"" Sep 06 14:14:24.581533 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.581511 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j"] Sep 06 14:14:24.592810 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.592784 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:24.630405 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.630379 2577 patch_prober.go:28] interesting pod/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 06 14:14:24.630504 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.630432 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" podUID="d13af05b-d79f-4675-b973-1ac87630cc57" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 06 14:14:24.691103 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.691065 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-podinfo\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.691103 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.691107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-credential-socket\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.691322 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.691139 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-token\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.691322 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.691229 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istiod-ca-cert\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.691322 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.691260 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-workload-socket\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.691435 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.691324 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-workload-certs\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.691435 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.691366 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97kqw\" (UniqueName: \"kubernetes.io/projected/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-kube-api-access-97kqw\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.691435 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.691399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-data\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.691435 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.691426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-envoy\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.720975 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.720935 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:24.792343 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.792306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-podinfo\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.792526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.792353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-credential-socket\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.792526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.792476 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-token\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.792658 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.792537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istiod-ca-cert\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.792658 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.792564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-workload-socket\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.792658 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.792615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-workload-certs\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.792658 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.792648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97kqw\" (UniqueName: \"kubernetes.io/projected/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-kube-api-access-97kqw\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.792871 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.792691 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-credential-socket\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.792871 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.792709 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-data\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.792871 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.792758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-envoy\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.793059 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.792868 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-workload-socket\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.793059 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.793040 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-workload-certs\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.793173 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.793047 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-data\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.793319 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.793300 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istiod-ca-cert\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.794900 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.794880 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-envoy\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.795036 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.795017 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-podinfo\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.800278 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.800259 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97kqw\" (UniqueName: \"kubernetes.io/projected/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-kube-api-access-97kqw\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.800421 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.800401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-token\") pod \"e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.819618 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.819556 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:24.880781 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.880757 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:24.924723 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:24.924685 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:25.024459 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:25.024422 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:25.152359 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:25.152331 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j"] Sep 06 14:14:25.154202 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:14:25.154179 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492ed6ed_6a8a_4c92_a8c8_40cdc530877c.slice/crio-5eaa9087e143b2d0bcfeb685aa9dcbf290df35396a38f69e35c3ee3651bc0cc0 WatchSource:0}: Error finding container 5eaa9087e143b2d0bcfeb685aa9dcbf290df35396a38f69e35c3ee3651bc0cc0: Status 404 returned error can't find the container with id 5eaa9087e143b2d0bcfeb685aa9dcbf290df35396a38f69e35c3ee3651bc0cc0 Sep 06 14:14:25.156359 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:25.156329 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:14:25.156454 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:25.156388 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:14:25.156454 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:25.156416 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:14:25.301417 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:25.301377 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" event={"ID":"492ed6ed-6a8a-4c92-a8c8-40cdc530877c","Type":"ContainerStarted","Data":"8030ac5615afa3152a7a3720332511246f60f64c79f2aba1e972d65eb983368e"} Sep 06 14:14:25.301607 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:25.301444 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" event={"ID":"492ed6ed-6a8a-4c92-a8c8-40cdc530877c","Type":"ContainerStarted","Data":"5eaa9087e143b2d0bcfeb685aa9dcbf290df35396a38f69e35c3ee3651bc0cc0"} Sep 06 14:14:25.323182 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:25.323124 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" podStartSLOduration=1.3231045 podStartE2EDuration="1.3231045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:14:25.320387299 +0000 UTC m=+996.775876161" watchObservedRunningTime="2026-09-06 14:14:25.3231045 +0000 UTC m=+996.778593366" Sep 06 14:14:25.584211 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:25.584171 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:25.685309 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:25.685276 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:25.788260 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:25.788230 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:25.881305 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:25.881228 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:26.284869 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:26.284837 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:26.885657 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:26.885629 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:27.312373 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:27.312340 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:27.313325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:27.313301 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:14:29.132574 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.132547 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:14:29.236852 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.236752 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13af05b-d79f-4675-b973-1ac87630cc57-istio-token\") pod \"d13af05b-d79f-4675-b973-1ac87630cc57\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " Sep 06 14:14:29.236852 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.236787 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-istio-envoy\") pod \"d13af05b-d79f-4675-b973-1ac87630cc57\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " Sep 06 14:14:29.236852 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.236817 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-workload-certs\") pod \"d13af05b-d79f-4675-b973-1ac87630cc57\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " Sep 06 14:14:29.236852 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.236852 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13af05b-d79f-4675-b973-1ac87630cc57-istiod-ca-cert\") pod \"d13af05b-d79f-4675-b973-1ac87630cc57\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " Sep 06 14:14:29.237240 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.236936 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-workload-socket\") pod \"d13af05b-d79f-4675-b973-1ac87630cc57\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " Sep 06 14:14:29.237240 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.236966 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-istio-data\") pod \"d13af05b-d79f-4675-b973-1ac87630cc57\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " Sep 06 14:14:29.237240 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.237037 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13af05b-d79f-4675-b973-1ac87630cc57-istio-podinfo\") pod \"d13af05b-d79f-4675-b973-1ac87630cc57\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " Sep 06 14:14:29.237240 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.237064 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-credential-socket\") pod \"d13af05b-d79f-4675-b973-1ac87630cc57\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " Sep 06 14:14:29.237240 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.237102 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r442\" (UniqueName: \"kubernetes.io/projected/d13af05b-d79f-4675-b973-1ac87630cc57-kube-api-access-9r442\") pod \"d13af05b-d79f-4675-b973-1ac87630cc57\" (UID: \"d13af05b-d79f-4675-b973-1ac87630cc57\") " Sep 06 14:14:29.237509 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.237209 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d13af05b-d79f-4675-b973-1ac87630cc57" (UID: "d13af05b-d79f-4675-b973-1ac87630cc57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:14:29.237509 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.237495 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-workload-certs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:29.237625 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.237504 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d13af05b-d79f-4675-b973-1ac87630cc57" (UID: "d13af05b-d79f-4675-b973-1ac87630cc57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:14:29.237625 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.237582 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-istio-data" (OuterVolumeSpecName: "istio-data") pod "d13af05b-d79f-4675-b973-1ac87630cc57" (UID: "d13af05b-d79f-4675-b973-1ac87630cc57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:14:29.237625 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.237612 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13af05b-d79f-4675-b973-1ac87630cc57-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d13af05b-d79f-4675-b973-1ac87630cc57" (UID: "d13af05b-d79f-4675-b973-1ac87630cc57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:14:29.237797 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.237726 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d13af05b-d79f-4675-b973-1ac87630cc57" (UID: "d13af05b-d79f-4675-b973-1ac87630cc57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:14:29.239069 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.239040 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d13af05b-d79f-4675-b973-1ac87630cc57" (UID: "d13af05b-d79f-4675-b973-1ac87630cc57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:14:29.239209 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.239191 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13af05b-d79f-4675-b973-1ac87630cc57-kube-api-access-9r442" (OuterVolumeSpecName: "kube-api-access-9r442") pod "d13af05b-d79f-4675-b973-1ac87630cc57" (UID: "d13af05b-d79f-4675-b973-1ac87630cc57"). InnerVolumeSpecName "kube-api-access-9r442". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:14:29.239478 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.239463 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13af05b-d79f-4675-b973-1ac87630cc57-istio-token" (OuterVolumeSpecName: "istio-token") pod "d13af05b-d79f-4675-b973-1ac87630cc57" (UID: "d13af05b-d79f-4675-b973-1ac87630cc57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:14:29.239531 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.239502 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d13af05b-d79f-4675-b973-1ac87630cc57-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d13af05b-d79f-4675-b973-1ac87630cc57" (UID: "d13af05b-d79f-4675-b973-1ac87630cc57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 14:14:29.323328 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.323292 2577 generic.go:356] "Generic (PLEG): container finished" podID="d13af05b-d79f-4675-b973-1ac87630cc57" containerID="035adb5478d6358b67e595dc412ab944fb327b1a37c78e64bc3a507146e1d5f1" exitCode=0 Sep 06 14:14:29.323488 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.323340 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" Sep 06 14:14:29.323488 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.323339 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" event={"ID":"d13af05b-d79f-4675-b973-1ac87630cc57","Type":"ContainerDied","Data":"035adb5478d6358b67e595dc412ab944fb327b1a37c78e64bc3a507146e1d5f1"} Sep 06 14:14:29.323488 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.323398 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq" event={"ID":"d13af05b-d79f-4675-b973-1ac87630cc57","Type":"ContainerDied","Data":"79c23a6c7c2ad6435c5c0daf25bab59ae69c30a67351b55a8ce327cef481c1f4"} Sep 06 14:14:29.323488 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.323424 2577 scope.go:117] "RemoveContainer" containerID="035adb5478d6358b67e595dc412ab944fb327b1a37c78e64bc3a507146e1d5f1" Sep 06 14:14:29.340735 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.340705 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-workload-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:29.340848 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.340741 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-istio-data\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:29.340848 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.340757 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13af05b-d79f-4675-b973-1ac87630cc57-istio-podinfo\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:29.340848 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.340769 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-credential-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:29.340848 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.340784 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r442\" (UniqueName: \"kubernetes.io/projected/d13af05b-d79f-4675-b973-1ac87630cc57-kube-api-access-9r442\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:29.340848 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.340798 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13af05b-d79f-4675-b973-1ac87630cc57-istio-token\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:29.340848 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.340811 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13af05b-d79f-4675-b973-1ac87630cc57-istio-envoy\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:29.340848 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.340823 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13af05b-d79f-4675-b973-1ac87630cc57-istiod-ca-cert\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:29.342888 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.342794 2577 scope.go:117] "RemoveContainer" containerID="76e6e77e7768b811258d9815d9a2b4e280d367bba8931c5d01a5e45a7a346bbd" Sep 06 14:14:29.346193 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.346149 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq"] Sep 06 14:14:29.349618 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.349597 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-dfca8b-openshift-default-6f7c66cdcb-b8mrq"] Sep 06 14:14:29.354189 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.354166 2577 scope.go:117] "RemoveContainer" containerID="035adb5478d6358b67e595dc412ab944fb327b1a37c78e64bc3a507146e1d5f1" Sep 06 14:14:29.354443 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:14:29.354422 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035adb5478d6358b67e595dc412ab944fb327b1a37c78e64bc3a507146e1d5f1\": container with ID starting with 035adb5478d6358b67e595dc412ab944fb327b1a37c78e64bc3a507146e1d5f1 not found: ID does not exist" containerID="035adb5478d6358b67e595dc412ab944fb327b1a37c78e64bc3a507146e1d5f1" Sep 06 14:14:29.354526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.354449 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035adb5478d6358b67e595dc412ab944fb327b1a37c78e64bc3a507146e1d5f1"} err="failed to get container status \"035adb5478d6358b67e595dc412ab944fb327b1a37c78e64bc3a507146e1d5f1\": rpc error: code = NotFound desc = could not find container \"035adb5478d6358b67e595dc412ab944fb327b1a37c78e64bc3a507146e1d5f1\": container with ID starting with 035adb5478d6358b67e595dc412ab944fb327b1a37c78e64bc3a507146e1d5f1 not found: ID does not exist" Sep 06 14:14:29.354526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.354467 2577 scope.go:117] "RemoveContainer" containerID="76e6e77e7768b811258d9815d9a2b4e280d367bba8931c5d01a5e45a7a346bbd" Sep 06 14:14:29.354726 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:14:29.354708 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e6e77e7768b811258d9815d9a2b4e280d367bba8931c5d01a5e45a7a346bbd\": container with ID starting with 76e6e77e7768b811258d9815d9a2b4e280d367bba8931c5d01a5e45a7a346bbd not found: ID does not exist" containerID="76e6e77e7768b811258d9815d9a2b4e280d367bba8931c5d01a5e45a7a346bbd" Sep 06 14:14:29.354790 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:29.354728 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e6e77e7768b811258d9815d9a2b4e280d367bba8931c5d01a5e45a7a346bbd"} err="failed to get container status \"76e6e77e7768b811258d9815d9a2b4e280d367bba8931c5d01a5e45a7a346bbd\": rpc error: code = NotFound desc = could not find container \"76e6e77e7768b811258d9815d9a2b4e280d367bba8931c5d01a5e45a7a346bbd\": container with ID starting with 76e6e77e7768b811258d9815d9a2b4e280d367bba8931c5d01a5e45a7a346bbd not found: ID does not exist" Sep 06 14:14:30.494422 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.494392 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:30.520622 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.520582 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt"] Sep 06 14:14:30.521037 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.521005 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" podUID="dbcb2df1-4343-4f25-9d05-a8c75df4f2eb" containerName="payload-processing" containerID="cri-o://e0d835de59f65c3cd0a9f843725e363eecabab0cbaae34510f3fb65e7bb53db6" gracePeriod=30 Sep 06 14:14:30.525846 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.525820 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht"] Sep 06 14:14:30.526080 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.526057 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" podUID="0191f47a-0637-42c6-abd3-eb7f02ece7c3" containerName="payload-pre-processing" containerID="cri-o://4f88b898df81ba4d1668a9122f966013a3f054a66ee3f741f710bf53b845de02" gracePeriod=30 Sep 06 14:14:30.560939 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.560899 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:30.623383 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.623339 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:30.655250 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.655205 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw"] Sep 06 14:14:30.657284 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.657256 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dfd6b-sr9vw"] Sep 06 14:14:30.685157 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.685123 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:30.809060 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.809039 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" Sep 06 14:14:30.812262 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.812243 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" Sep 06 14:14:30.955606 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.955576 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqjsq\" (UniqueName: \"kubernetes.io/projected/dbcb2df1-4343-4f25-9d05-a8c75df4f2eb-kube-api-access-nqjsq\") pod \"dbcb2df1-4343-4f25-9d05-a8c75df4f2eb\" (UID: \"dbcb2df1-4343-4f25-9d05-a8c75df4f2eb\") " Sep 06 14:14:30.955931 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.955852 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0191f47a-0637-42c6-abd3-eb7f02ece7c3-plugins-config-volume\") pod \"0191f47a-0637-42c6-abd3-eb7f02ece7c3\" (UID: \"0191f47a-0637-42c6-abd3-eb7f02ece7c3\") " Sep 06 14:14:30.956050 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.955941 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4bcp\" (UniqueName: \"kubernetes.io/projected/0191f47a-0637-42c6-abd3-eb7f02ece7c3-kube-api-access-n4bcp\") pod \"0191f47a-0637-42c6-abd3-eb7f02ece7c3\" (UID: \"0191f47a-0637-42c6-abd3-eb7f02ece7c3\") " Sep 06 14:14:30.956050 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.955987 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbcb2df1-4343-4f25-9d05-a8c75df4f2eb-plugins-config-volume\") pod \"dbcb2df1-4343-4f25-9d05-a8c75df4f2eb\" (UID: \"dbcb2df1-4343-4f25-9d05-a8c75df4f2eb\") " Sep 06 14:14:30.956545 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.956513 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0191f47a-0637-42c6-abd3-eb7f02ece7c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0191f47a-0637-42c6-abd3-eb7f02ece7c3" (UID: "0191f47a-0637-42c6-abd3-eb7f02ece7c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:14:30.956703 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.956677 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbcb2df1-4343-4f25-9d05-a8c75df4f2eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dbcb2df1-4343-4f25-9d05-a8c75df4f2eb" (UID: "dbcb2df1-4343-4f25-9d05-a8c75df4f2eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:14:30.958486 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.958462 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbcb2df1-4343-4f25-9d05-a8c75df4f2eb-kube-api-access-nqjsq" (OuterVolumeSpecName: "kube-api-access-nqjsq") pod "dbcb2df1-4343-4f25-9d05-a8c75df4f2eb" (UID: "dbcb2df1-4343-4f25-9d05-a8c75df4f2eb"). InnerVolumeSpecName "kube-api-access-nqjsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:14:30.958573 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.958517 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0191f47a-0637-42c6-abd3-eb7f02ece7c3-kube-api-access-n4bcp" (OuterVolumeSpecName: "kube-api-access-n4bcp") pod "0191f47a-0637-42c6-abd3-eb7f02ece7c3" (UID: "0191f47a-0637-42c6-abd3-eb7f02ece7c3"). InnerVolumeSpecName "kube-api-access-n4bcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:14:30.996176 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:30.996147 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:31.057381 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.057306 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0191f47a-0637-42c6-abd3-eb7f02ece7c3-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:31.057381 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.057334 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4bcp\" (UniqueName: \"kubernetes.io/projected/0191f47a-0637-42c6-abd3-eb7f02ece7c3-kube-api-access-n4bcp\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:31.057381 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.057347 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbcb2df1-4343-4f25-9d05-a8c75df4f2eb-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:31.057381 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.057365 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqjsq\" (UniqueName: \"kubernetes.io/projected/dbcb2df1-4343-4f25-9d05-a8c75df4f2eb-kube-api-access-nqjsq\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:31.067867 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.067838 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae55c260-20c1-49be-a82b-c52a0f07a370" path="/var/lib/kubelet/pods/ae55c260-20c1-49be-a82b-c52a0f07a370/volumes" Sep 06 14:14:31.068317 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.068303 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d13af05b-d79f-4675-b973-1ac87630cc57" path="/var/lib/kubelet/pods/d13af05b-d79f-4675-b973-1ac87630cc57/volumes" Sep 06 14:14:31.337040 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.336947 2577 generic.go:356] "Generic (PLEG): container finished" podID="0191f47a-0637-42c6-abd3-eb7f02ece7c3" containerID="4f88b898df81ba4d1668a9122f966013a3f054a66ee3f741f710bf53b845de02" exitCode=0 Sep 06 14:14:31.337176 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.337053 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" Sep 06 14:14:31.337176 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.337100 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" event={"ID":"0191f47a-0637-42c6-abd3-eb7f02ece7c3","Type":"ContainerDied","Data":"4f88b898df81ba4d1668a9122f966013a3f054a66ee3f741f710bf53b845de02"} Sep 06 14:14:31.337176 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.337154 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht" event={"ID":"0191f47a-0637-42c6-abd3-eb7f02ece7c3","Type":"ContainerDied","Data":"dfcad1e2bff738e265981be12efde5f89b0bab14fdada89a252eb4363e5475dd"} Sep 06 14:14:31.337324 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.337184 2577 scope.go:117] "RemoveContainer" containerID="4f88b898df81ba4d1668a9122f966013a3f054a66ee3f741f710bf53b845de02" Sep 06 14:14:31.338790 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.338763 2577 generic.go:356] "Generic (PLEG): container finished" podID="dbcb2df1-4343-4f25-9d05-a8c75df4f2eb" containerID="e0d835de59f65c3cd0a9f843725e363eecabab0cbaae34510f3fb65e7bb53db6" exitCode=0 Sep 06 14:14:31.338947 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.338803 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" Sep 06 14:14:31.338947 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.338802 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" event={"ID":"dbcb2df1-4343-4f25-9d05-a8c75df4f2eb","Type":"ContainerDied","Data":"e0d835de59f65c3cd0a9f843725e363eecabab0cbaae34510f3fb65e7bb53db6"} Sep 06 14:14:31.338947 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.338859 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt" event={"ID":"dbcb2df1-4343-4f25-9d05-a8c75df4f2eb","Type":"ContainerDied","Data":"b82ec21cca10ed289302b61203d03fd078c01b58798dece38b306707afdeb554"} Sep 06 14:14:31.350072 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.350050 2577 scope.go:117] "RemoveContainer" containerID="4f88b898df81ba4d1668a9122f966013a3f054a66ee3f741f710bf53b845de02" Sep 06 14:14:31.350375 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:14:31.350354 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f88b898df81ba4d1668a9122f966013a3f054a66ee3f741f710bf53b845de02\": container with ID starting with 4f88b898df81ba4d1668a9122f966013a3f054a66ee3f741f710bf53b845de02 not found: ID does not exist" containerID="4f88b898df81ba4d1668a9122f966013a3f054a66ee3f741f710bf53b845de02" Sep 06 14:14:31.350472 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.350381 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f88b898df81ba4d1668a9122f966013a3f054a66ee3f741f710bf53b845de02"} err="failed to get container status \"4f88b898df81ba4d1668a9122f966013a3f054a66ee3f741f710bf53b845de02\": rpc error: code = NotFound desc = could not find container \"4f88b898df81ba4d1668a9122f966013a3f054a66ee3f741f710bf53b845de02\": container with ID starting with 4f88b898df81ba4d1668a9122f966013a3f054a66ee3f741f710bf53b845de02 not found: ID does not exist" Sep 06 14:14:31.350472 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.350402 2577 scope.go:117] "RemoveContainer" containerID="e0d835de59f65c3cd0a9f843725e363eecabab0cbaae34510f3fb65e7bb53db6" Sep 06 14:14:31.356031 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.356008 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt"] Sep 06 14:14:31.359008 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.358991 2577 scope.go:117] "RemoveContainer" containerID="e0d835de59f65c3cd0a9f843725e363eecabab0cbaae34510f3fb65e7bb53db6" Sep 06 14:14:31.359303 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:14:31.359260 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0d835de59f65c3cd0a9f843725e363eecabab0cbaae34510f3fb65e7bb53db6\": container with ID starting with e0d835de59f65c3cd0a9f843725e363eecabab0cbaae34510f3fb65e7bb53db6 not found: ID does not exist" containerID="e0d835de59f65c3cd0a9f843725e363eecabab0cbaae34510f3fb65e7bb53db6" Sep 06 14:14:31.359357 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.359315 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d835de59f65c3cd0a9f843725e363eecabab0cbaae34510f3fb65e7bb53db6"} err="failed to get container status \"e0d835de59f65c3cd0a9f843725e363eecabab0cbaae34510f3fb65e7bb53db6\": rpc error: code = NotFound desc = could not find container \"e0d835de59f65c3cd0a9f843725e363eecabab0cbaae34510f3fb65e7bb53db6\": container with ID starting with e0d835de59f65c3cd0a9f843725e363eecabab0cbaae34510f3fb65e7bb53db6 not found: ID does not exist" Sep 06 14:14:31.359357 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.359268 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dfd6b-9489786bc-l9rkt"] Sep 06 14:14:31.368289 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.368257 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht"] Sep 06 14:14:31.370222 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:31.370203 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dfd6b-d4bfc87c8-tncht"] Sep 06 14:14:32.072611 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:32.072576 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:32.129972 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:32.129946 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:32.717005 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:32.716956 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:33.073702 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:33.073669 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0191f47a-0637-42c6-abd3-eb7f02ece7c3" path="/var/lib/kubelet/pods/0191f47a-0637-42c6-abd3-eb7f02ece7c3/volumes" Sep 06 14:14:33.074058 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:33.074043 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbcb2df1-4343-4f25-9d05-a8c75df4f2eb" path="/var/lib/kubelet/pods/dbcb2df1-4343-4f25-9d05-a8c75df4f2eb/volumes" Sep 06 14:14:33.295149 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:33.295121 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:14:36.620739 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:36.620704 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:36.720637 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:36.720604 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:36.862766 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:36.862732 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:37.317498 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:37.317458 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:38.575332 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:38.575300 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:38.632071 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:38.632040 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:38.801539 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:38.801508 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:39.096766 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:39.096725 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:39.405098 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:39.405013 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:41.709642 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:41.709607 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:41.807493 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:41.807455 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:42.847898 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:42.847862 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:42.903831 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:42.903798 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:43.150678 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.150597 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:43.321638 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.321597 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:43.421131 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.421052 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:43.422573 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.422533 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx"] Sep 06 14:14:43.423029 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.423012 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbcb2df1-4343-4f25-9d05-a8c75df4f2eb" containerName="payload-processing" Sep 06 14:14:43.423137 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.423031 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbcb2df1-4343-4f25-9d05-a8c75df4f2eb" containerName="payload-processing" Sep 06 14:14:43.423137 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.423069 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d13af05b-d79f-4675-b973-1ac87630cc57" containerName="istio-proxy" Sep 06 14:14:43.423137 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.423078 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d13af05b-d79f-4675-b973-1ac87630cc57" containerName="istio-proxy" Sep 06 14:14:43.423137 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.423087 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d13af05b-d79f-4675-b973-1ac87630cc57" containerName="istio-proxy" Sep 06 14:14:43.423137 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.423095 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d13af05b-d79f-4675-b973-1ac87630cc57" containerName="istio-proxy" Sep 06 14:14:43.423137 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.423113 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0191f47a-0637-42c6-abd3-eb7f02ece7c3" containerName="payload-pre-processing" Sep 06 14:14:43.423137 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.423121 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0191f47a-0637-42c6-abd3-eb7f02ece7c3" containerName="payload-pre-processing" Sep 06 14:14:43.423462 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.423218 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbcb2df1-4343-4f25-9d05-a8c75df4f2eb" containerName="payload-processing" Sep 06 14:14:43.423462 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.423238 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d13af05b-d79f-4675-b973-1ac87630cc57" containerName="istio-proxy" Sep 06 14:14:43.423462 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.423251 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0191f47a-0637-42c6-abd3-eb7f02ece7c3" containerName="payload-pre-processing" Sep 06 14:14:43.423462 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.423261 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d13af05b-d79f-4675-b973-1ac87630cc57" containerName="istio-proxy" Sep 06 14:14:43.426501 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.426483 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" Sep 06 14:14:43.429193 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.429168 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-81fae671-dockercfg-l9rtx\"" Sep 06 14:14:43.429193 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.429192 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-81fae671\"" Sep 06 14:14:43.437663 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.437643 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx"] Sep 06 14:14:43.461093 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.461072 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z"] Sep 06 14:14:43.463763 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.463748 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" Sep 06 14:14:43.473967 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.473936 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z"] Sep 06 14:14:43.572665 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.572623 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632b501-8254-497b-b12e-91e305f2a0c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx\" (UID: \"0632b501-8254-497b-b12e-91e305f2a0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" Sep 06 14:14:43.572854 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.572676 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8z76\" (UniqueName: \"kubernetes.io/projected/6f90524e-2080-4f72-8564-d5967b39bcb0-kube-api-access-z8z76\") pod \"payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z\" (UID: \"6f90524e-2080-4f72-8564-d5967b39bcb0\") " pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" Sep 06 14:14:43.572854 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.572736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsdj5\" (UniqueName: \"kubernetes.io/projected/0632b501-8254-497b-b12e-91e305f2a0c4-kube-api-access-lsdj5\") pod \"payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx\" (UID: \"0632b501-8254-497b-b12e-91e305f2a0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" Sep 06 14:14:43.572854 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.572824 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f90524e-2080-4f72-8564-d5967b39bcb0-plugins-config-volume\") pod \"payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z\" (UID: \"6f90524e-2080-4f72-8564-d5967b39bcb0\") " pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" Sep 06 14:14:43.674024 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.673936 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632b501-8254-497b-b12e-91e305f2a0c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx\" (UID: \"0632b501-8254-497b-b12e-91e305f2a0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" Sep 06 14:14:43.674024 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.673973 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8z76\" (UniqueName: \"kubernetes.io/projected/6f90524e-2080-4f72-8564-d5967b39bcb0-kube-api-access-z8z76\") pod \"payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z\" (UID: \"6f90524e-2080-4f72-8564-d5967b39bcb0\") " pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" Sep 06 14:14:43.674024 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.673998 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsdj5\" (UniqueName: \"kubernetes.io/projected/0632b501-8254-497b-b12e-91e305f2a0c4-kube-api-access-lsdj5\") pod \"payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx\" (UID: \"0632b501-8254-497b-b12e-91e305f2a0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" Sep 06 14:14:43.674290 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.674044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f90524e-2080-4f72-8564-d5967b39bcb0-plugins-config-volume\") pod \"payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z\" (UID: \"6f90524e-2080-4f72-8564-d5967b39bcb0\") " pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" Sep 06 14:14:43.674730 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.674702 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632b501-8254-497b-b12e-91e305f2a0c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx\" (UID: \"0632b501-8254-497b-b12e-91e305f2a0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" Sep 06 14:14:43.674809 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.674761 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f90524e-2080-4f72-8564-d5967b39bcb0-plugins-config-volume\") pod \"payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z\" (UID: \"6f90524e-2080-4f72-8564-d5967b39bcb0\") " pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" Sep 06 14:14:43.681154 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.681135 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsdj5\" (UniqueName: \"kubernetes.io/projected/0632b501-8254-497b-b12e-91e305f2a0c4-kube-api-access-lsdj5\") pod \"payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx\" (UID: \"0632b501-8254-497b-b12e-91e305f2a0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" Sep 06 14:14:43.681466 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.681445 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8z76\" (UniqueName: \"kubernetes.io/projected/6f90524e-2080-4f72-8564-d5967b39bcb0-kube-api-access-z8z76\") pod \"payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z\" (UID: \"6f90524e-2080-4f72-8564-d5967b39bcb0\") " pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" Sep 06 14:14:43.736878 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.736852 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" Sep 06 14:14:43.774950 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:43.774916 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" Sep 06 14:14:44.005279 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:44.005232 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx"] Sep 06 14:14:44.007533 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:14:44.007505 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0632b501_8254_497b_b12e_91e305f2a0c4.slice/crio-edba6c6b185dcf1897cac3f486e035c66671261386a7c52d71ed6157b30c97c8 WatchSource:0}: Error finding container edba6c6b185dcf1897cac3f486e035c66671261386a7c52d71ed6157b30c97c8: Status 404 returned error can't find the container with id edba6c6b185dcf1897cac3f486e035c66671261386a7c52d71ed6157b30c97c8 Sep 06 14:14:44.054360 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:44.054335 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z"] Sep 06 14:14:44.055973 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:14:44.055951 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f90524e_2080_4f72_8564_d5967b39bcb0.slice/crio-dc446f1d0fa6a2ac151fbd6108ddf62985907a830c0d8a2f9560152323156c0f WatchSource:0}: Error finding container dc446f1d0fa6a2ac151fbd6108ddf62985907a830c0d8a2f9560152323156c0f: Status 404 returned error can't find the container with id dc446f1d0fa6a2ac151fbd6108ddf62985907a830c0d8a2f9560152323156c0f Sep 06 14:14:44.413826 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:44.413791 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" event={"ID":"6f90524e-2080-4f72-8564-d5967b39bcb0","Type":"ContainerStarted","Data":"22e82a8afefdf5b1b21f35c330db920dd62391f43855d9b85b7cdaf5b71a7a10"} Sep 06 14:14:44.414082 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:44.413843 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" event={"ID":"6f90524e-2080-4f72-8564-d5967b39bcb0","Type":"ContainerStarted","Data":"dc446f1d0fa6a2ac151fbd6108ddf62985907a830c0d8a2f9560152323156c0f"} Sep 06 14:14:44.414082 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:44.413947 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" Sep 06 14:14:44.415478 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:44.415458 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" event={"ID":"0632b501-8254-497b-b12e-91e305f2a0c4","Type":"ContainerStarted","Data":"a8edd5016bc924e8caa3cf2a385a92ee403a3a102762cc8cb3f99965345cfada"} Sep 06 14:14:44.415558 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:44.415496 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" event={"ID":"0632b501-8254-497b-b12e-91e305f2a0c4","Type":"ContainerStarted","Data":"edba6c6b185dcf1897cac3f486e035c66671261386a7c52d71ed6157b30c97c8"} Sep 06 14:14:44.415558 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:44.415522 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" Sep 06 14:14:44.429956 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:44.429892 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" podStartSLOduration=1.429874551 podStartE2EDuration="1.429874551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:14:44.427895678 +0000 UTC m=+1015.883384541" watchObservedRunningTime="2026-09-06 14:14:44.429874551 +0000 UTC m=+1015.885363415" Sep 06 14:14:44.441552 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:44.441501 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" podStartSLOduration=1.4414848679999999 podStartE2EDuration="1.441484868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:14:44.441357557 +0000 UTC m=+1015.896852420" watchObservedRunningTime="2026-09-06 14:14:44.441484868 +0000 UTC m=+1015.896973732" Sep 06 14:14:46.091321 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:46.091285 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:14:47.310318 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:47.310282 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:47.925266 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:47.925234 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:47.954114 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:47.954079 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s"] Sep 06 14:14:47.954318 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:47.954289 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" podUID="1e110340-3eed-4d79-8191-bc66e1549897" containerName="payload-pre-processing" containerID="cri-o://206bb4b9eeaf9f3024121b29bf68774a75c3eb3c695c8fad45d3e02c22548697" gracePeriod=30 Sep 06 14:14:48.029321 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.029297 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:48.088824 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.088792 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:48.139862 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.139829 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:48.225877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.225848 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" Sep 06 14:14:48.314244 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.314213 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh8zj\" (UniqueName: \"kubernetes.io/projected/1e110340-3eed-4d79-8191-bc66e1549897-kube-api-access-fh8zj\") pod \"1e110340-3eed-4d79-8191-bc66e1549897\" (UID: \"1e110340-3eed-4d79-8191-bc66e1549897\") " Sep 06 14:14:48.314610 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.314278 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e110340-3eed-4d79-8191-bc66e1549897-plugins-config-volume\") pod \"1e110340-3eed-4d79-8191-bc66e1549897\" (UID: \"1e110340-3eed-4d79-8191-bc66e1549897\") " Sep 06 14:14:48.314708 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.314670 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e110340-3eed-4d79-8191-bc66e1549897-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1e110340-3eed-4d79-8191-bc66e1549897" (UID: "1e110340-3eed-4d79-8191-bc66e1549897"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:14:48.316344 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.316319 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e110340-3eed-4d79-8191-bc66e1549897-kube-api-access-fh8zj" (OuterVolumeSpecName: "kube-api-access-fh8zj") pod "1e110340-3eed-4d79-8191-bc66e1549897" (UID: "1e110340-3eed-4d79-8191-bc66e1549897"). InnerVolumeSpecName "kube-api-access-fh8zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:14:48.415273 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.415235 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh8zj\" (UniqueName: \"kubernetes.io/projected/1e110340-3eed-4d79-8191-bc66e1549897-kube-api-access-fh8zj\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:48.415273 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.415267 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e110340-3eed-4d79-8191-bc66e1549897-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:14:48.432326 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.432297 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:48.439984 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.439958 2577 generic.go:356] "Generic (PLEG): container finished" podID="1e110340-3eed-4d79-8191-bc66e1549897" containerID="206bb4b9eeaf9f3024121b29bf68774a75c3eb3c695c8fad45d3e02c22548697" exitCode=0 Sep 06 14:14:48.440096 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.439998 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" Sep 06 14:14:48.440096 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.440048 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" event={"ID":"1e110340-3eed-4d79-8191-bc66e1549897","Type":"ContainerDied","Data":"206bb4b9eeaf9f3024121b29bf68774a75c3eb3c695c8fad45d3e02c22548697"} Sep 06 14:14:48.440161 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.440097 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s" event={"ID":"1e110340-3eed-4d79-8191-bc66e1549897","Type":"ContainerDied","Data":"05822eb0a95210be56591a6865d8b4b317fe89a911656d7b89145a095e3937a5"} Sep 06 14:14:48.440161 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.440119 2577 scope.go:117] "RemoveContainer" containerID="206bb4b9eeaf9f3024121b29bf68774a75c3eb3c695c8fad45d3e02c22548697" Sep 06 14:14:48.449121 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.448988 2577 scope.go:117] "RemoveContainer" containerID="206bb4b9eeaf9f3024121b29bf68774a75c3eb3c695c8fad45d3e02c22548697" Sep 06 14:14:48.449253 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:14:48.449233 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206bb4b9eeaf9f3024121b29bf68774a75c3eb3c695c8fad45d3e02c22548697\": container with ID starting with 206bb4b9eeaf9f3024121b29bf68774a75c3eb3c695c8fad45d3e02c22548697 not found: ID does not exist" containerID="206bb4b9eeaf9f3024121b29bf68774a75c3eb3c695c8fad45d3e02c22548697" Sep 06 14:14:48.449301 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.449263 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206bb4b9eeaf9f3024121b29bf68774a75c3eb3c695c8fad45d3e02c22548697"} err="failed to get container status \"206bb4b9eeaf9f3024121b29bf68774a75c3eb3c695c8fad45d3e02c22548697\": rpc error: code = NotFound desc = could not find container \"206bb4b9eeaf9f3024121b29bf68774a75c3eb3c695c8fad45d3e02c22548697\": container with ID starting with 206bb4b9eeaf9f3024121b29bf68774a75c3eb3c695c8fad45d3e02c22548697 not found: ID does not exist" Sep 06 14:14:48.460241 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.460186 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s"] Sep 06 14:14:48.466424 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:48.466404 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ec585-79dc8765db-wgr2s"] Sep 06 14:14:49.068476 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:49.068443 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e110340-3eed-4d79-8191-bc66e1549897" path="/var/lib/kubelet/pods/1e110340-3eed-4d79-8191-bc66e1549897/volumes" Sep 06 14:14:52.807135 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:52.807102 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:53.996582 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:53.996540 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:54.096366 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:54.096338 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:54.229337 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:54.229306 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:54.780446 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:54.780408 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:55.061748 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:55.061660 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:55.263714 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:55.263677 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:55.425602 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:55.425517 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" Sep 06 14:14:55.425602 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:55.425592 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" Sep 06 14:14:56.003523 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:56.003488 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:14:57.846151 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:57.846104 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:57.944427 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:57.944397 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:59.102181 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.102147 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:59.203141 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.203109 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:59.467134 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.467053 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:59.626214 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.626182 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:59.721342 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.721268 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:14:59.725150 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.725124 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh"] Sep 06 14:14:59.725621 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.725600 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e110340-3eed-4d79-8191-bc66e1549897" containerName="payload-pre-processing" Sep 06 14:14:59.725621 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.725616 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e110340-3eed-4d79-8191-bc66e1549897" containerName="payload-pre-processing" Sep 06 14:14:59.725807 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.725695 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e110340-3eed-4d79-8191-bc66e1549897" containerName="payload-pre-processing" Sep 06 14:14:59.729350 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.729333 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" Sep 06 14:14:59.731679 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.731660 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-0aaff3\"" Sep 06 14:14:59.731746 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.731696 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-0aaff3-dockercfg-tw8lw\"" Sep 06 14:14:59.739053 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.738832 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh"] Sep 06 14:14:59.820289 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.820253 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7da9281-f964-4eae-a5a7-c2cab274009a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh\" (UID: \"f7da9281-f964-4eae-a5a7-c2cab274009a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" Sep 06 14:14:59.820474 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.820303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqtg\" (UniqueName: \"kubernetes.io/projected/f7da9281-f964-4eae-a5a7-c2cab274009a-kube-api-access-cvqtg\") pod \"payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh\" (UID: \"f7da9281-f964-4eae-a5a7-c2cab274009a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" Sep 06 14:14:59.921456 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.921420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7da9281-f964-4eae-a5a7-c2cab274009a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh\" (UID: \"f7da9281-f964-4eae-a5a7-c2cab274009a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" Sep 06 14:14:59.921456 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.921457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqtg\" (UniqueName: \"kubernetes.io/projected/f7da9281-f964-4eae-a5a7-c2cab274009a-kube-api-access-cvqtg\") pod \"payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh\" (UID: \"f7da9281-f964-4eae-a5a7-c2cab274009a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" Sep 06 14:14:59.922196 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.922172 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7da9281-f964-4eae-a5a7-c2cab274009a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh\" (UID: \"f7da9281-f964-4eae-a5a7-c2cab274009a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" Sep 06 14:14:59.928778 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:14:59.928747 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqtg\" (UniqueName: \"kubernetes.io/projected/f7da9281-f964-4eae-a5a7-c2cab274009a-kube-api-access-cvqtg\") pod \"payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh\" (UID: \"f7da9281-f964-4eae-a5a7-c2cab274009a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" Sep 06 14:15:00.041739 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:00.041706 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" Sep 06 14:15:00.133364 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:00.133323 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-81fae671-29811735-67vnp"] Sep 06 14:15:00.136638 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:00.136611 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-81fae671-29811735-67vnp" Sep 06 14:15:00.138803 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:00.138782 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\"" Sep 06 14:15:00.143580 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:00.143522 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-81fae671-29811735-67vnp"] Sep 06 14:15:00.150264 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:00.150248 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-81fae671-29811735-67vnp" Sep 06 14:15:00.307533 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:00.307510 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh"] Sep 06 14:15:00.309165 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:15:00.309138 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7da9281_f964_4eae_a5a7_c2cab274009a.slice/crio-d2df82319b56d4e55a18e607d802063d330e2b7bc6fd2c1161cd9982a811513e WatchSource:0}: Error finding container d2df82319b56d4e55a18e607d802063d330e2b7bc6fd2c1161cd9982a811513e: Status 404 returned error can't find the container with id d2df82319b56d4e55a18e607d802063d330e2b7bc6fd2c1161cd9982a811513e Sep 06 14:15:00.417750 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:00.417720 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-81fae671-29811735-67vnp"] Sep 06 14:15:00.421997 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:15:00.421974 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0e8121_d29f_4f19_a494_c74acbe702ce.slice/crio-9b4c09cf899eecaa51bc0b86e0f6a0788fa9d1b09815436808f95f7b8a2b0ce4 WatchSource:0}: Error finding container 9b4c09cf899eecaa51bc0b86e0f6a0788fa9d1b09815436808f95f7b8a2b0ce4: Status 404 returned error can't find the container with id 9b4c09cf899eecaa51bc0b86e0f6a0788fa9d1b09815436808f95f7b8a2b0ce4 Sep 06 14:15:00.503696 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:00.503670 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-81fae671-29811735-67vnp" event={"ID":"2a0e8121-d29f-4f19-a494-c74acbe702ce","Type":"ContainerStarted","Data":"9b4c09cf899eecaa51bc0b86e0f6a0788fa9d1b09815436808f95f7b8a2b0ce4"} Sep 06 14:15:00.505157 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:00.505135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" event={"ID":"f7da9281-f964-4eae-a5a7-c2cab274009a","Type":"ContainerStarted","Data":"d9911fa92dcb5e86be35db635dce217c9931cfbc50321e48428660b9dbfde4d5"} Sep 06 14:15:00.505250 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:00.505174 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" event={"ID":"f7da9281-f964-4eae-a5a7-c2cab274009a","Type":"ContainerStarted","Data":"d2df82319b56d4e55a18e607d802063d330e2b7bc6fd2c1161cd9982a811513e"} Sep 06 14:15:00.505291 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:00.505268 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" Sep 06 14:15:00.520709 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:00.520663 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" podStartSLOduration=1.5206487819999999 podStartE2EDuration="1.520648782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:15:00.51885391 +0000 UTC m=+1031.974342774" watchObservedRunningTime="2026-09-06 14:15:00.520648782 +0000 UTC m=+1031.976137638" Sep 06 14:15:01.512247 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:01.512166 2577 generic.go:356] "Generic (PLEG): container finished" podID="2a0e8121-d29f-4f19-a494-c74acbe702ce" containerID="f266057070026c11586c21949613fdcb41baf7299819a8742f2d727b42392a6b" exitCode=0 Sep 06 14:15:01.512586 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:01.512250 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-81fae671-29811735-67vnp" event={"ID":"2a0e8121-d29f-4f19-a494-c74acbe702ce","Type":"ContainerDied","Data":"f266057070026c11586c21949613fdcb41baf7299819a8742f2d727b42392a6b"} Sep 06 14:15:02.652935 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:02.652889 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-81fae671-29811735-67vnp" Sep 06 14:15:03.526488 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:03.526457 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-81fae671-29811735-67vnp" Sep 06 14:15:03.526669 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:03.526455 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-81fae671-29811735-67vnp" event={"ID":"2a0e8121-d29f-4f19-a494-c74acbe702ce","Type":"ContainerDied","Data":"9b4c09cf899eecaa51bc0b86e0f6a0788fa9d1b09815436808f95f7b8a2b0ce4"} Sep 06 14:15:03.526669 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:03.526592 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b4c09cf899eecaa51bc0b86e0f6a0788fa9d1b09815436808f95f7b8a2b0ce4" Sep 06 14:15:04.379407 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:04.379373 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:04.479423 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:04.479393 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:04.582495 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:04.582466 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:06.599965 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:06.599934 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:07.218382 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:07.218346 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:07.316408 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:07.316377 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:07.622354 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:07.622321 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:10.092069 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:10.092035 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:15:11.514737 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:11.514705 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" Sep 06 14:15:13.405556 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:13.405525 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:13.455591 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:13.455562 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:13.733130 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:13.733041 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81fae671-w8nsx"] Sep 06 14:15:13.733573 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:13.733542 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a0e8121-d29f-4f19-a494-c74acbe702ce" containerName="cleanup" Sep 06 14:15:13.733573 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:13.733566 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a0e8121-d29f-4f19-a494-c74acbe702ce" containerName="cleanup" Sep 06 14:15:13.733725 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:13.733648 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a0e8121-d29f-4f19-a494-c74acbe702ce" containerName="cleanup" Sep 06 14:15:13.735974 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:13.735956 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81fae671-w8nsx" Sep 06 14:15:13.738304 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:13.738278 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 14:15:13.738422 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:13.738338 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\"" Sep 06 14:15:13.742347 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:13.742327 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81fae671-w8nsx"] Sep 06 14:15:13.858091 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:13.858063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05fa717a-95eb-43c0-a912-6786f718f5c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-81fae671-w8nsx\" (UID: \"05fa717a-95eb-43c0-a912-6786f718f5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81fae671-w8nsx" Sep 06 14:15:13.959200 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:13.959174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05fa717a-95eb-43c0-a912-6786f718f5c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-81fae671-w8nsx\" (UID: \"05fa717a-95eb-43c0-a912-6786f718f5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81fae671-w8nsx" Sep 06 14:15:13.959755 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:13.959735 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05fa717a-95eb-43c0-a912-6786f718f5c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-81fae671-w8nsx\" (UID: \"05fa717a-95eb-43c0-a912-6786f718f5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81fae671-w8nsx" Sep 06 14:15:14.049876 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:14.049854 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81fae671-w8nsx" Sep 06 14:15:14.321958 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:14.320124 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81fae671-w8nsx"] Sep 06 14:15:14.328184 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:15:14.328149 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fa717a_95eb_43c0_a912_6786f718f5c6.slice/crio-67a58281024701e9789a6bbf9ad559bffb94efae83207e50a113754857c031e4 WatchSource:0}: Error finding container 67a58281024701e9789a6bbf9ad559bffb94efae83207e50a113754857c031e4: Status 404 returned error can't find the container with id 67a58281024701e9789a6bbf9ad559bffb94efae83207e50a113754857c031e4 Sep 06 14:15:14.587670 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:14.587603 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81fae671-w8nsx" event={"ID":"05fa717a-95eb-43c0-a912-6786f718f5c6","Type":"ContainerStarted","Data":"67a58281024701e9789a6bbf9ad559bffb94efae83207e50a113754857c031e4"} Sep 06 14:15:15.593960 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:15.593926 2577 generic.go:356] "Generic (PLEG): container finished" podID="05fa717a-95eb-43c0-a912-6786f718f5c6" containerID="4ba6e23e4dc2a1e99765008beaef42b6121ba0dc8e37938f86264adb4855a8ad" exitCode=0 Sep 06 14:15:15.594338 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:15.594000 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81fae671-w8nsx" event={"ID":"05fa717a-95eb-43c0-a912-6786f718f5c6","Type":"ContainerDied","Data":"4ba6e23e4dc2a1e99765008beaef42b6121ba0dc8e37938f86264adb4855a8ad"} Sep 06 14:15:16.736509 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:16.736484 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81fae671-w8nsx" Sep 06 14:15:16.888463 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:16.888389 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05fa717a-95eb-43c0-a912-6786f718f5c6-maas-api-service-ca\") pod \"05fa717a-95eb-43c0-a912-6786f718f5c6\" (UID: \"05fa717a-95eb-43c0-a912-6786f718f5c6\") " Sep 06 14:15:16.888723 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:16.888702 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05fa717a-95eb-43c0-a912-6786f718f5c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05fa717a-95eb-43c0-a912-6786f718f5c6" (UID: "05fa717a-95eb-43c0-a912-6786f718f5c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:15:16.989502 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:16.989467 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05fa717a-95eb-43c0-a912-6786f718f5c6-maas-api-service-ca\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:15:17.605625 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:17.605596 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81fae671-w8nsx" Sep 06 14:15:17.605625 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:17.605611 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81fae671-w8nsx" event={"ID":"05fa717a-95eb-43c0-a912-6786f718f5c6","Type":"ContainerDied","Data":"67a58281024701e9789a6bbf9ad559bffb94efae83207e50a113754857c031e4"} Sep 06 14:15:17.605838 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:17.605657 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67a58281024701e9789a6bbf9ad559bffb94efae83207e50a113754857c031e4" Sep 06 14:15:18.823582 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:18.823543 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:18.907262 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:18.907227 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j"] Sep 06 14:15:18.907527 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:18.907495 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" podUID="492ed6ed-6a8a-4c92-a8c8-40cdc530877c" containerName="istio-proxy" containerID="cri-o://8030ac5615afa3152a7a3720332511246f60f64c79f2aba1e972d65eb983368e" gracePeriod=30 Sep 06 14:15:18.934621 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:18.934593 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:19.147983 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:19.147880 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:19.245775 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:19.245721 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:20.209596 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:20.209555 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:22.736266 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:22.736231 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:23.041741 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:23.041709 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:23.340565 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:23.340482 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:23.627089 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:23.627010 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:23.860359 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:23.860326 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:24.156297 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.156275 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:15:24.261997 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.261921 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-podinfo\") pod \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " Sep 06 14:15:24.262154 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.261995 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-token\") pod \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " Sep 06 14:15:24.262154 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.262023 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-credential-socket\") pod \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " Sep 06 14:15:24.262154 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.262047 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97kqw\" (UniqueName: \"kubernetes.io/projected/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-kube-api-access-97kqw\") pod \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " Sep 06 14:15:24.262154 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.262079 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-workload-certs\") pod \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " Sep 06 14:15:24.262154 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.262124 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-data\") pod \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " Sep 06 14:15:24.262412 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.262164 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-envoy\") pod \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " Sep 06 14:15:24.262412 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.262263 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-workload-socket\") pod \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " Sep 06 14:15:24.262412 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.262292 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istiod-ca-cert\") pod \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\" (UID: \"492ed6ed-6a8a-4c92-a8c8-40cdc530877c\") " Sep 06 14:15:24.263465 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.263016 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "492ed6ed-6a8a-4c92-a8c8-40cdc530877c" (UID: "492ed6ed-6a8a-4c92-a8c8-40cdc530877c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:15:24.263465 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.262769 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-data" (OuterVolumeSpecName: "istio-data") pod "492ed6ed-6a8a-4c92-a8c8-40cdc530877c" (UID: "492ed6ed-6a8a-4c92-a8c8-40cdc530877c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:15:24.263465 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.263093 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "492ed6ed-6a8a-4c92-a8c8-40cdc530877c" (UID: "492ed6ed-6a8a-4c92-a8c8-40cdc530877c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:15:24.263465 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.263119 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "492ed6ed-6a8a-4c92-a8c8-40cdc530877c" (UID: "492ed6ed-6a8a-4c92-a8c8-40cdc530877c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:15:24.263465 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.263391 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "492ed6ed-6a8a-4c92-a8c8-40cdc530877c" (UID: "492ed6ed-6a8a-4c92-a8c8-40cdc530877c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:15:24.265612 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.265295 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-token" (OuterVolumeSpecName: "istio-token") pod "492ed6ed-6a8a-4c92-a8c8-40cdc530877c" (UID: "492ed6ed-6a8a-4c92-a8c8-40cdc530877c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:15:24.266537 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.266498 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-kube-api-access-97kqw" (OuterVolumeSpecName: "kube-api-access-97kqw") pod "492ed6ed-6a8a-4c92-a8c8-40cdc530877c" (UID: "492ed6ed-6a8a-4c92-a8c8-40cdc530877c"). InnerVolumeSpecName "kube-api-access-97kqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:15:24.266666 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.266537 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "492ed6ed-6a8a-4c92-a8c8-40cdc530877c" (UID: "492ed6ed-6a8a-4c92-a8c8-40cdc530877c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:15:24.266879 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.266855 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "492ed6ed-6a8a-4c92-a8c8-40cdc530877c" (UID: "492ed6ed-6a8a-4c92-a8c8-40cdc530877c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 14:15:24.322053 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.322024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:24.364117 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.364091 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-workload-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:15:24.364246 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.364121 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istiod-ca-cert\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:15:24.364246 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.364134 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-podinfo\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:15:24.364246 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.364146 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-token\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:15:24.364246 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.364158 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-credential-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:15:24.364246 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.364171 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97kqw\" (UniqueName: \"kubernetes.io/projected/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-kube-api-access-97kqw\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:15:24.364246 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.364184 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-workload-certs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:15:24.364246 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.364198 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-data\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:15:24.364246 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.364211 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492ed6ed-6a8a-4c92-a8c8-40cdc530877c-istio-envoy\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:15:24.406966 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.406927 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-81fae671-29811735-67vnp"] Sep 06 14:15:24.408698 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.408672 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:24.409382 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.409358 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-81fae671-29811735-67vnp"] Sep 06 14:15:24.430895 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.430861 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z"] Sep 06 14:15:24.431098 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.431074 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" podUID="6f90524e-2080-4f72-8564-d5967b39bcb0" containerName="payload-processing" containerID="cri-o://22e82a8afefdf5b1b21f35c330db920dd62391f43855d9b85b7cdaf5b71a7a10" gracePeriod=30 Sep 06 14:15:24.441265 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.440595 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx"] Sep 06 14:15:24.441265 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.440801 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" podUID="0632b501-8254-497b-b12e-91e305f2a0c4" containerName="payload-pre-processing" containerID="cri-o://a8edd5016bc924e8caa3cf2a385a92ee403a3a102762cc8cb3f99965345cfada" gracePeriod=30 Sep 06 14:15:24.471474 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.471442 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:24.570703 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.570673 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:24.588564 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.588524 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81fae671-w8nsx"] Sep 06 14:15:24.588725 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.588594 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81fae671-w8nsx"] Sep 06 14:15:24.630381 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.630351 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:24.651295 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.651256 2577 generic.go:356] "Generic (PLEG): container finished" podID="492ed6ed-6a8a-4c92-a8c8-40cdc530877c" containerID="8030ac5615afa3152a7a3720332511246f60f64c79f2aba1e972d65eb983368e" exitCode=0 Sep 06 14:15:24.651451 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.651332 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" Sep 06 14:15:24.651451 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.651333 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" event={"ID":"492ed6ed-6a8a-4c92-a8c8-40cdc530877c","Type":"ContainerDied","Data":"8030ac5615afa3152a7a3720332511246f60f64c79f2aba1e972d65eb983368e"} Sep 06 14:15:24.651451 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.651419 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j" event={"ID":"492ed6ed-6a8a-4c92-a8c8-40cdc530877c","Type":"ContainerDied","Data":"5eaa9087e143b2d0bcfeb685aa9dcbf290df35396a38f69e35c3ee3651bc0cc0"} Sep 06 14:15:24.651451 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.651444 2577 scope.go:117] "RemoveContainer" containerID="8030ac5615afa3152a7a3720332511246f60f64c79f2aba1e972d65eb983368e" Sep 06 14:15:24.653654 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.653627 2577 generic.go:356] "Generic (PLEG): container finished" podID="6f90524e-2080-4f72-8564-d5967b39bcb0" containerID="22e82a8afefdf5b1b21f35c330db920dd62391f43855d9b85b7cdaf5b71a7a10" exitCode=0 Sep 06 14:15:24.660884 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.653719 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" event={"ID":"6f90524e-2080-4f72-8564-d5967b39bcb0","Type":"ContainerDied","Data":"22e82a8afefdf5b1b21f35c330db920dd62391f43855d9b85b7cdaf5b71a7a10"} Sep 06 14:15:24.666487 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.666456 2577 generic.go:356] "Generic (PLEG): container finished" podID="0632b501-8254-497b-b12e-91e305f2a0c4" containerID="a8edd5016bc924e8caa3cf2a385a92ee403a3a102762cc8cb3f99965345cfada" exitCode=0 Sep 06 14:15:24.666721 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.666520 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" event={"ID":"0632b501-8254-497b-b12e-91e305f2a0c4","Type":"ContainerDied","Data":"a8edd5016bc924e8caa3cf2a385a92ee403a3a102762cc8cb3f99965345cfada"} Sep 06 14:15:24.667338 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.667313 2577 scope.go:117] "RemoveContainer" containerID="8030ac5615afa3152a7a3720332511246f60f64c79f2aba1e972d65eb983368e" Sep 06 14:15:24.667814 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:15:24.667789 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8030ac5615afa3152a7a3720332511246f60f64c79f2aba1e972d65eb983368e\": container with ID starting with 8030ac5615afa3152a7a3720332511246f60f64c79f2aba1e972d65eb983368e not found: ID does not exist" containerID="8030ac5615afa3152a7a3720332511246f60f64c79f2aba1e972d65eb983368e" Sep 06 14:15:24.668056 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.668032 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8030ac5615afa3152a7a3720332511246f60f64c79f2aba1e972d65eb983368e"} err="failed to get container status \"8030ac5615afa3152a7a3720332511246f60f64c79f2aba1e972d65eb983368e\": rpc error: code = NotFound desc = could not find container \"8030ac5615afa3152a7a3720332511246f60f64c79f2aba1e972d65eb983368e\": container with ID starting with 8030ac5615afa3152a7a3720332511246f60f64c79f2aba1e972d65eb983368e not found: ID does not exist" Sep 06 14:15:24.687286 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.687072 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j"] Sep 06 14:15:24.694257 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.694230 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-4e801f-openshift-default-69d6564b68-fqd8j"] Sep 06 14:15:24.706500 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.706477 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" Sep 06 14:15:24.713349 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.713323 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" Sep 06 14:15:24.869496 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.869409 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8z76\" (UniqueName: \"kubernetes.io/projected/6f90524e-2080-4f72-8564-d5967b39bcb0-kube-api-access-z8z76\") pod \"6f90524e-2080-4f72-8564-d5967b39bcb0\" (UID: \"6f90524e-2080-4f72-8564-d5967b39bcb0\") " Sep 06 14:15:24.869496 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.869467 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsdj5\" (UniqueName: \"kubernetes.io/projected/0632b501-8254-497b-b12e-91e305f2a0c4-kube-api-access-lsdj5\") pod \"0632b501-8254-497b-b12e-91e305f2a0c4\" (UID: \"0632b501-8254-497b-b12e-91e305f2a0c4\") " Sep 06 14:15:24.869496 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.869497 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632b501-8254-497b-b12e-91e305f2a0c4-plugins-config-volume\") pod \"0632b501-8254-497b-b12e-91e305f2a0c4\" (UID: \"0632b501-8254-497b-b12e-91e305f2a0c4\") " Sep 06 14:15:24.870064 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.869562 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f90524e-2080-4f72-8564-d5967b39bcb0-plugins-config-volume\") pod \"6f90524e-2080-4f72-8564-d5967b39bcb0\" (UID: \"6f90524e-2080-4f72-8564-d5967b39bcb0\") " Sep 06 14:15:24.870064 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.869947 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0632b501-8254-497b-b12e-91e305f2a0c4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0632b501-8254-497b-b12e-91e305f2a0c4" (UID: "0632b501-8254-497b-b12e-91e305f2a0c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:15:24.870064 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.869947 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f90524e-2080-4f72-8564-d5967b39bcb0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f90524e-2080-4f72-8564-d5967b39bcb0" (UID: "6f90524e-2080-4f72-8564-d5967b39bcb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:15:24.871577 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.871556 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0632b501-8254-497b-b12e-91e305f2a0c4-kube-api-access-lsdj5" (OuterVolumeSpecName: "kube-api-access-lsdj5") pod "0632b501-8254-497b-b12e-91e305f2a0c4" (UID: "0632b501-8254-497b-b12e-91e305f2a0c4"). InnerVolumeSpecName "kube-api-access-lsdj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:15:24.871654 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.871603 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f90524e-2080-4f72-8564-d5967b39bcb0-kube-api-access-z8z76" (OuterVolumeSpecName: "kube-api-access-z8z76") pod "6f90524e-2080-4f72-8564-d5967b39bcb0" (UID: "6f90524e-2080-4f72-8564-d5967b39bcb0"). InnerVolumeSpecName "kube-api-access-z8z76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:15:24.917409 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.917374 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:24.970755 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.970725 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f90524e-2080-4f72-8564-d5967b39bcb0-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:15:24.970755 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.970750 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8z76\" (UniqueName: \"kubernetes.io/projected/6f90524e-2080-4f72-8564-d5967b39bcb0-kube-api-access-z8z76\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:15:24.970934 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.970772 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsdj5\" (UniqueName: \"kubernetes.io/projected/0632b501-8254-497b-b12e-91e305f2a0c4-kube-api-access-lsdj5\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:15:24.970934 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:24.970781 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632b501-8254-497b-b12e-91e305f2a0c4-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:15:25.068009 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:25.067981 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05fa717a-95eb-43c0-a912-6786f718f5c6" path="/var/lib/kubelet/pods/05fa717a-95eb-43c0-a912-6786f718f5c6/volumes" Sep 06 14:15:25.068290 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:25.068277 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a0e8121-d29f-4f19-a494-c74acbe702ce" path="/var/lib/kubelet/pods/2a0e8121-d29f-4f19-a494-c74acbe702ce/volumes" Sep 06 14:15:25.068626 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:25.068614 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="492ed6ed-6a8a-4c92-a8c8-40cdc530877c" path="/var/lib/kubelet/pods/492ed6ed-6a8a-4c92-a8c8-40cdc530877c/volumes" Sep 06 14:15:25.673066 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:25.673039 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" Sep 06 14:15:25.673066 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:25.673051 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z" event={"ID":"6f90524e-2080-4f72-8564-d5967b39bcb0","Type":"ContainerDied","Data":"dc446f1d0fa6a2ac151fbd6108ddf62985907a830c0d8a2f9560152323156c0f"} Sep 06 14:15:25.673320 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:25.673117 2577 scope.go:117] "RemoveContainer" containerID="22e82a8afefdf5b1b21f35c330db920dd62391f43855d9b85b7cdaf5b71a7a10" Sep 06 14:15:25.674461 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:25.674443 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" Sep 06 14:15:25.674580 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:25.674490 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx" event={"ID":"0632b501-8254-497b-b12e-91e305f2a0c4","Type":"ContainerDied","Data":"edba6c6b185dcf1897cac3f486e035c66671261386a7c52d71ed6157b30c97c8"} Sep 06 14:15:25.682106 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:25.682086 2577 scope.go:117] "RemoveContainer" containerID="a8edd5016bc924e8caa3cf2a385a92ee403a3a102762cc8cb3f99965345cfada" Sep 06 14:15:25.693653 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:25.693631 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z"] Sep 06 14:15:25.696569 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:25.696546 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81fae671-57ffddf489-4hk8z"] Sep 06 14:15:25.705158 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:25.705139 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx"] Sep 06 14:15:25.708886 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:25.708862 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81fae671-77968c8578-rl7jx"] Sep 06 14:15:26.002070 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:26.001992 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:26.101050 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:26.101018 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:27.068737 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.068707 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0632b501-8254-497b-b12e-91e305f2a0c4" path="/var/lib/kubelet/pods/0632b501-8254-497b-b12e-91e305f2a0c4/volumes" Sep 06 14:15:27.069104 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.069089 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f90524e-2080-4f72-8564-d5967b39bcb0" path="/var/lib/kubelet/pods/6f90524e-2080-4f72-8564-d5967b39bcb0/volumes" Sep 06 14:15:27.088859 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.088829 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:15:27.242878 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.242795 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:27.323387 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.323349 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:27.709140 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.709108 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:27.811370 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.811342 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:27.814529 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.814504 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2"] Sep 06 14:15:27.814929 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.814916 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f90524e-2080-4f72-8564-d5967b39bcb0" containerName="payload-processing" Sep 06 14:15:27.815016 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.814931 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f90524e-2080-4f72-8564-d5967b39bcb0" containerName="payload-processing" Sep 06 14:15:27.815016 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.814941 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05fa717a-95eb-43c0-a912-6786f718f5c6" containerName="revoke-keys" Sep 06 14:15:27.815016 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.814946 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="05fa717a-95eb-43c0-a912-6786f718f5c6" containerName="revoke-keys" Sep 06 14:15:27.815016 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.814962 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0632b501-8254-497b-b12e-91e305f2a0c4" containerName="payload-pre-processing" Sep 06 14:15:27.815016 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.814968 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0632b501-8254-497b-b12e-91e305f2a0c4" containerName="payload-pre-processing" Sep 06 14:15:27.815016 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.814975 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492ed6ed-6a8a-4c92-a8c8-40cdc530877c" containerName="istio-proxy" Sep 06 14:15:27.815016 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.814980 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="492ed6ed-6a8a-4c92-a8c8-40cdc530877c" containerName="istio-proxy" Sep 06 14:15:27.815234 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.815063 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="05fa717a-95eb-43c0-a912-6786f718f5c6" containerName="revoke-keys" Sep 06 14:15:27.815234 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.815077 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0632b501-8254-497b-b12e-91e305f2a0c4" containerName="payload-pre-processing" Sep 06 14:15:27.815234 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.815087 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="492ed6ed-6a8a-4c92-a8c8-40cdc530877c" containerName="istio-proxy" Sep 06 14:15:27.815234 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.815096 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f90524e-2080-4f72-8564-d5967b39bcb0" containerName="payload-processing" Sep 06 14:15:27.818172 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.818156 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" Sep 06 14:15:27.820470 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.820453 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c443d8-dockercfg-g7snc\"" Sep 06 14:15:27.820558 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.820459 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c443d8\"" Sep 06 14:15:27.841068 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.841034 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2"] Sep 06 14:15:27.843539 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.843513 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq"] Sep 06 14:15:27.846105 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.846084 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" Sep 06 14:15:27.850826 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.850802 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq"] Sep 06 14:15:27.882360 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.882323 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:27.997783 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.997697 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fnws\" (UniqueName: \"kubernetes.io/projected/0608cdb9-87d3-4a8d-9c52-e5246d8b38e2-kube-api-access-4fnws\") pod \"payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2\" (UID: \"0608cdb9-87d3-4a8d-9c52-e5246d8b38e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" Sep 06 14:15:27.997783 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.997768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea60091-a515-422f-8cde-42894f870611-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq\" (UID: \"9ea60091-a515-422f-8cde-42894f870611\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" Sep 06 14:15:27.998009 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.997798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0608cdb9-87d3-4a8d-9c52-e5246d8b38e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2\" (UID: \"0608cdb9-87d3-4a8d-9c52-e5246d8b38e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" Sep 06 14:15:27.998009 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:27.997898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6spc\" (UniqueName: \"kubernetes.io/projected/9ea60091-a515-422f-8cde-42894f870611-kube-api-access-q6spc\") pod \"payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq\" (UID: \"9ea60091-a515-422f-8cde-42894f870611\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" Sep 06 14:15:28.098637 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.098602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6spc\" (UniqueName: \"kubernetes.io/projected/9ea60091-a515-422f-8cde-42894f870611-kube-api-access-q6spc\") pod \"payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq\" (UID: \"9ea60091-a515-422f-8cde-42894f870611\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" Sep 06 14:15:28.099114 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.098693 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fnws\" (UniqueName: \"kubernetes.io/projected/0608cdb9-87d3-4a8d-9c52-e5246d8b38e2-kube-api-access-4fnws\") pod \"payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2\" (UID: \"0608cdb9-87d3-4a8d-9c52-e5246d8b38e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" Sep 06 14:15:28.099114 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.098722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea60091-a515-422f-8cde-42894f870611-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq\" (UID: \"9ea60091-a515-422f-8cde-42894f870611\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" Sep 06 14:15:28.099114 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.098739 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0608cdb9-87d3-4a8d-9c52-e5246d8b38e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2\" (UID: \"0608cdb9-87d3-4a8d-9c52-e5246d8b38e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" Sep 06 14:15:28.099424 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.099404 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0608cdb9-87d3-4a8d-9c52-e5246d8b38e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2\" (UID: \"0608cdb9-87d3-4a8d-9c52-e5246d8b38e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" Sep 06 14:15:28.099460 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.099423 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea60091-a515-422f-8cde-42894f870611-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq\" (UID: \"9ea60091-a515-422f-8cde-42894f870611\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" Sep 06 14:15:28.106551 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.106527 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6spc\" (UniqueName: \"kubernetes.io/projected/9ea60091-a515-422f-8cde-42894f870611-kube-api-access-q6spc\") pod \"payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq\" (UID: \"9ea60091-a515-422f-8cde-42894f870611\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" Sep 06 14:15:28.106674 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.106578 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fnws\" (UniqueName: \"kubernetes.io/projected/0608cdb9-87d3-4a8d-9c52-e5246d8b38e2-kube-api-access-4fnws\") pod \"payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2\" (UID: \"0608cdb9-87d3-4a8d-9c52-e5246d8b38e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" Sep 06 14:15:28.130429 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.130400 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" Sep 06 14:15:28.158621 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.158592 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" Sep 06 14:15:28.403457 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.403433 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2"] Sep 06 14:15:28.405665 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:15:28.405640 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0608cdb9_87d3_4a8d_9c52_e5246d8b38e2.slice/crio-ae41734a1f35a91da699f54c1cca61e5d38527a4d33eefd210abaea1d84ddb18 WatchSource:0}: Error finding container ae41734a1f35a91da699f54c1cca61e5d38527a4d33eefd210abaea1d84ddb18: Status 404 returned error can't find the container with id ae41734a1f35a91da699f54c1cca61e5d38527a4d33eefd210abaea1d84ddb18 Sep 06 14:15:28.437961 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.437939 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq"] Sep 06 14:15:28.440060 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:15:28.440032 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea60091_a515_422f_8cde_42894f870611.slice/crio-d2319ed3b58bd33218ab277c07d75d92c3a4c684fc8e38aeacad6a5aaba9cf03 WatchSource:0}: Error finding container d2319ed3b58bd33218ab277c07d75d92c3a4c684fc8e38aeacad6a5aaba9cf03: Status 404 returned error can't find the container with id d2319ed3b58bd33218ab277c07d75d92c3a4c684fc8e38aeacad6a5aaba9cf03 Sep 06 14:15:28.700174 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.700070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" event={"ID":"0608cdb9-87d3-4a8d-9c52-e5246d8b38e2","Type":"ContainerStarted","Data":"0e86e0b8a3a026b1fbf0f3738b0331771f9aeb85bd7a8c721d307d58f60a220c"} Sep 06 14:15:28.700174 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.700129 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" event={"ID":"0608cdb9-87d3-4a8d-9c52-e5246d8b38e2","Type":"ContainerStarted","Data":"ae41734a1f35a91da699f54c1cca61e5d38527a4d33eefd210abaea1d84ddb18"} Sep 06 14:15:28.700174 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.700159 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" Sep 06 14:15:28.701533 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.701507 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" event={"ID":"9ea60091-a515-422f-8cde-42894f870611","Type":"ContainerStarted","Data":"eeb492cf7fea75b6e5407ab22cac4c8183134da2831a8d639b2765cedbf96402"} Sep 06 14:15:28.701607 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.701557 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" event={"ID":"9ea60091-a515-422f-8cde-42894f870611","Type":"ContainerStarted","Data":"d2319ed3b58bd33218ab277c07d75d92c3a4c684fc8e38aeacad6a5aaba9cf03"} Sep 06 14:15:28.701607 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.701586 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" Sep 06 14:15:28.715442 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.715390 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" podStartSLOduration=1.715377584 podStartE2EDuration="1.715377584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:15:28.712817112 +0000 UTC m=+1060.168305977" watchObservedRunningTime="2026-09-06 14:15:28.715377584 +0000 UTC m=+1060.170866440" Sep 06 14:15:28.727312 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:28.727260 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" podStartSLOduration=1.727247515 podStartE2EDuration="1.727247515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:15:28.726211579 +0000 UTC m=+1060.181700441" watchObservedRunningTime="2026-09-06 14:15:28.727247515 +0000 UTC m=+1060.182736378" Sep 06 14:15:30.566656 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:30.566626 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:30.623709 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:30.623681 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:30.801467 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:30.801437 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:31.131856 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:31.131821 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:31.231259 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:31.231223 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:31.332303 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:31.332267 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:37.139133 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:37.139099 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:37.200142 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:37.200092 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:37.275094 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:37.275061 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:37.375344 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:37.375302 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:37.477400 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:37.477322 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:39.388961 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:39.388927 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:15:39.708147 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:39.708066 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" Sep 06 14:15:39.708462 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:39.708443 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" Sep 06 14:15:42.936525 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:42.936444 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:42.990027 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:42.989996 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:43.796672 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:43.796639 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:44.087352 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:44.087267 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:44.402272 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:44.402198 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:47.092396 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:47.092361 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:47.192132 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:47.192097 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:48.593503 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.593466 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d"] Sep 06 14:15:48.597038 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.597016 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:48.599107 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.599087 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-6328f9-serving-cert\"" Sep 06 14:15:48.599260 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.599243 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 06 14:15:48.600223 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.600201 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4lj84\"" Sep 06 14:15:48.601039 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.601017 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 06 14:15:48.601140 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.601056 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 14:15:48.603564 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.603541 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d"] Sep 06 14:15:48.616264 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.616237 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:48.687368 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.687339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32edbadd-fcf9-4db2-946b-6e95600645eb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:48.687482 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.687393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32edbadd-fcf9-4db2-946b-6e95600645eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:48.687482 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.687414 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs6ln\" (UniqueName: \"kubernetes.io/projected/32edbadd-fcf9-4db2-946b-6e95600645eb-kube-api-access-zs6ln\") pod \"maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:48.687482 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.687443 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32edbadd-fcf9-4db2-946b-6e95600645eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:48.715437 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.715414 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:48.788491 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.788456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32edbadd-fcf9-4db2-946b-6e95600645eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:48.788671 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.788496 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs6ln\" (UniqueName: \"kubernetes.io/projected/32edbadd-fcf9-4db2-946b-6e95600645eb-kube-api-access-zs6ln\") pod \"maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:48.788671 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.788551 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32edbadd-fcf9-4db2-946b-6e95600645eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:48.788671 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:15:48.788592 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-serving-cert: secret "maas-api-e2e-shared-a-w2-6328f9-serving-cert" not found Sep 06 14:15:48.788671 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:15:48.788658 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32edbadd-fcf9-4db2-946b-6e95600645eb-maas-api-tls podName:32edbadd-fcf9-4db2-946b-6e95600645eb nodeName:}" failed. No retries permitted until 2026-09-06 14:15:49.28863622 +0000 UTC m=+1080.744125075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32edbadd-fcf9-4db2-946b-6e95600645eb-maas-api-tls") pod "maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" (UID: "32edbadd-fcf9-4db2-946b-6e95600645eb") : secret "maas-api-e2e-shared-a-w2-6328f9-serving-cert" not found Sep 06 14:15:48.788671 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.788671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32edbadd-fcf9-4db2-946b-6e95600645eb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:48.789336 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.789307 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32edbadd-fcf9-4db2-946b-6e95600645eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:48.791448 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.791427 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32edbadd-fcf9-4db2-946b-6e95600645eb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:48.796387 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:48.796365 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs6ln\" (UniqueName: \"kubernetes.io/projected/32edbadd-fcf9-4db2-946b-6e95600645eb-kube-api-access-zs6ln\") pod \"maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:49.073325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.073292 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:49.153497 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.153463 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:49.153982 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.153958 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7"] Sep 06 14:15:49.156752 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.156733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" Sep 06 14:15:49.158978 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.158955 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-6328f9\"" Sep 06 14:15:49.159090 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.159064 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-6328f9-dockercfg-fkp29\"" Sep 06 14:15:49.165309 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.165289 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7"] Sep 06 14:15:49.200159 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.200138 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr"] Sep 06 14:15:49.203742 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.203715 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" Sep 06 14:15:49.211535 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.211510 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr"] Sep 06 14:15:49.284022 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.283994 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:49.293847 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.293826 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f8e593b-f6f4-475e-9327-00454b3af81a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7\" (UID: \"2f8e593b-f6f4-475e-9327-00454b3af81a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" Sep 06 14:15:49.293966 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.293863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32edbadd-fcf9-4db2-946b-6e95600645eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:49.293966 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.293897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpc9j\" (UniqueName: \"kubernetes.io/projected/2f8e593b-f6f4-475e-9327-00454b3af81a-kube-api-access-cpc9j\") pod \"payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7\" (UID: \"2f8e593b-f6f4-475e-9327-00454b3af81a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" Sep 06 14:15:49.293966 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.293959 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb4890a-ffaf-4cc2-a224-c1a54bd98461-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr\" (UID: \"ffb4890a-ffaf-4cc2-a224-c1a54bd98461\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" Sep 06 14:15:49.294109 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.293992 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbsts\" (UniqueName: \"kubernetes.io/projected/ffb4890a-ffaf-4cc2-a224-c1a54bd98461-kube-api-access-cbsts\") pod \"payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr\" (UID: \"ffb4890a-ffaf-4cc2-a224-c1a54bd98461\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" Sep 06 14:15:49.296249 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.296218 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32edbadd-fcf9-4db2-946b-6e95600645eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:49.395118 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.395046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpc9j\" (UniqueName: \"kubernetes.io/projected/2f8e593b-f6f4-475e-9327-00454b3af81a-kube-api-access-cpc9j\") pod \"payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7\" (UID: \"2f8e593b-f6f4-475e-9327-00454b3af81a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" Sep 06 14:15:49.395118 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.395086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb4890a-ffaf-4cc2-a224-c1a54bd98461-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr\" (UID: \"ffb4890a-ffaf-4cc2-a224-c1a54bd98461\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" Sep 06 14:15:49.395288 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.395121 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbsts\" (UniqueName: \"kubernetes.io/projected/ffb4890a-ffaf-4cc2-a224-c1a54bd98461-kube-api-access-cbsts\") pod \"payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr\" (UID: \"ffb4890a-ffaf-4cc2-a224-c1a54bd98461\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" Sep 06 14:15:49.395288 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.395190 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f8e593b-f6f4-475e-9327-00454b3af81a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7\" (UID: \"2f8e593b-f6f4-475e-9327-00454b3af81a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" Sep 06 14:15:49.395757 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.395737 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb4890a-ffaf-4cc2-a224-c1a54bd98461-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr\" (UID: \"ffb4890a-ffaf-4cc2-a224-c1a54bd98461\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" Sep 06 14:15:49.395883 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.395865 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f8e593b-f6f4-475e-9327-00454b3af81a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7\" (UID: \"2f8e593b-f6f4-475e-9327-00454b3af81a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" Sep 06 14:15:49.402065 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.402044 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpc9j\" (UniqueName: \"kubernetes.io/projected/2f8e593b-f6f4-475e-9327-00454b3af81a-kube-api-access-cpc9j\") pod \"payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7\" (UID: \"2f8e593b-f6f4-475e-9327-00454b3af81a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" Sep 06 14:15:49.402278 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.402260 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbsts\" (UniqueName: \"kubernetes.io/projected/ffb4890a-ffaf-4cc2-a224-c1a54bd98461-kube-api-access-cbsts\") pod \"payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr\" (UID: \"ffb4890a-ffaf-4cc2-a224-c1a54bd98461\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" Sep 06 14:15:49.419162 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.419136 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:49.468310 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.468285 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" Sep 06 14:15:49.514587 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.514497 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4lj84\"" Sep 06 14:15:49.523420 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.523148 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:49.530643 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.530606 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" Sep 06 14:15:49.737503 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.737480 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7"] Sep 06 14:15:49.739263 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:15:49.739235 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8e593b_f6f4_475e_9327_00454b3af81a.slice/crio-9014730d96db2112a2be31ef4f958c3d1f49e0d9c48d124f7c7f79391d93c5d1 WatchSource:0}: Error finding container 9014730d96db2112a2be31ef4f958c3d1f49e0d9c48d124f7c7f79391d93c5d1: Status 404 returned error can't find the container with id 9014730d96db2112a2be31ef4f958c3d1f49e0d9c48d124f7c7f79391d93c5d1 Sep 06 14:15:49.807439 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.807150 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr"] Sep 06 14:15:49.808705 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:15:49.808598 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb4890a_ffaf_4cc2_a224_c1a54bd98461.slice/crio-7af374e89988fad5bd14e40cada85cb1b1ceb648db31b6a6505539a9215d6782 WatchSource:0}: Error finding container 7af374e89988fad5bd14e40cada85cb1b1ceb648db31b6a6505539a9215d6782: Status 404 returned error can't find the container with id 7af374e89988fad5bd14e40cada85cb1b1ceb648db31b6a6505539a9215d6782 Sep 06 14:15:49.810384 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.810355 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d"] Sep 06 14:15:49.810852 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:15:49.810832 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32edbadd_fcf9_4db2_946b_6e95600645eb.slice/crio-1080dab1b9dff7ecedc7f6221c414876c76485822daa62f25ea4962a48704036 WatchSource:0}: Error finding container 1080dab1b9dff7ecedc7f6221c414876c76485822daa62f25ea4962a48704036: Status 404 returned error can't find the container with id 1080dab1b9dff7ecedc7f6221c414876c76485822daa62f25ea4962a48704036 Sep 06 14:15:49.812456 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.812437 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 14:15:49.815531 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.815507 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" event={"ID":"2f8e593b-f6f4-475e-9327-00454b3af81a","Type":"ContainerStarted","Data":"4802057e59e6e68e20aa5521c767e2fad6cf58f678b91581bd4af5f331910b44"} Sep 06 14:15:49.815636 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.815554 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" event={"ID":"2f8e593b-f6f4-475e-9327-00454b3af81a","Type":"ContainerStarted","Data":"9014730d96db2112a2be31ef4f958c3d1f49e0d9c48d124f7c7f79391d93c5d1"} Sep 06 14:15:49.815636 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.815580 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" Sep 06 14:15:49.833488 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:49.833432 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" podStartSLOduration=0.833411464 podStartE2EDuration="833.411464ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:15:49.831448221 +0000 UTC m=+1081.286937084" watchObservedRunningTime="2026-09-06 14:15:49.833411464 +0000 UTC m=+1081.288900324" Sep 06 14:15:50.116361 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:50.116323 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:50.824671 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:50.824633 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" event={"ID":"32edbadd-fcf9-4db2-946b-6e95600645eb","Type":"ContainerStarted","Data":"1080dab1b9dff7ecedc7f6221c414876c76485822daa62f25ea4962a48704036"} Sep 06 14:15:50.828685 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:50.828656 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" event={"ID":"ffb4890a-ffaf-4cc2-a224-c1a54bd98461","Type":"ContainerStarted","Data":"50fa21284df2cd8f3d76e4601305972d47366c174118ba477d422c5d26411449"} Sep 06 14:15:50.828855 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:50.828702 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" event={"ID":"ffb4890a-ffaf-4cc2-a224-c1a54bd98461","Type":"ContainerStarted","Data":"7af374e89988fad5bd14e40cada85cb1b1ceb648db31b6a6505539a9215d6782"} Sep 06 14:15:50.828855 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:50.828744 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" Sep 06 14:15:50.846293 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:50.846236 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" podStartSLOduration=1.846220717 podStartE2EDuration="1.846220717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:15:50.842881805 +0000 UTC m=+1082.298370667" watchObservedRunningTime="2026-09-06 14:15:50.846220717 +0000 UTC m=+1082.301709579" Sep 06 14:15:50.891102 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:50.891075 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:52.841585 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:52.841546 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" event={"ID":"32edbadd-fcf9-4db2-946b-6e95600645eb","Type":"ContainerStarted","Data":"ef6aa86a4267e061e6410dfb0315b43c508f45fbfa6703bbe7ddaef657f83ca5"} Sep 06 14:15:52.841996 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:52.841646 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:52.859078 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:52.859030 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" podStartSLOduration=2.453287874 podStartE2EDuration="4.859011012s" totalImagesPullingTime="2.405723138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:15:52.856517597 +0000 UTC m=+1084.312006624" watchObservedRunningTime="2026-09-06 14:15:52.859011012 +0000 UTC m=+1084.314499887" Sep 06 14:15:53.621404 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:53.621373 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:15:53.942584 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:53.942494 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:15:58.852440 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:58.852410 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:15:59.237943 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:15:59.237831 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:00.831223 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:00.831189 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" Sep 06 14:16:01.032509 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:01.032472 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:02.843657 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:02.843626 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" Sep 06 14:16:04.686114 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:04.686082 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:16:12.001295 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:12.001264 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:12.309642 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:12.309608 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:12.573079 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:12.572998 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:12.802346 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:12.802307 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:15.266986 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:15.266938 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:15.365668 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:15.365633 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:16.430184 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:16.430154 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:16.529166 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:16.529128 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:16.777964 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:16.777931 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:16.962901 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:16.962871 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:17.063959 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.063866 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:17.068517 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.068492 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc"] Sep 06 14:16:17.072255 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.072234 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" Sep 06 14:16:17.076525 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.076498 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a388e1-dockercfg-wvzjc\"" Sep 06 14:16:17.076739 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.076719 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a388e1\"" Sep 06 14:16:17.085360 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.085332 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc"] Sep 06 14:16:17.162852 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.162810 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49vbm\" (UniqueName: \"kubernetes.io/projected/a8aade61-f68c-4205-bed5-13acf52493e9-kube-api-access-49vbm\") pod \"payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc\" (UID: \"a8aade61-f68c-4205-bed5-13acf52493e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" Sep 06 14:16:17.163182 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.163153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8aade61-f68c-4205-bed5-13acf52493e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc\" (UID: \"a8aade61-f68c-4205-bed5-13acf52493e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" Sep 06 14:16:17.264210 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.264176 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8aade61-f68c-4205-bed5-13acf52493e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc\" (UID: \"a8aade61-f68c-4205-bed5-13acf52493e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" Sep 06 14:16:17.264402 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.264219 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49vbm\" (UniqueName: \"kubernetes.io/projected/a8aade61-f68c-4205-bed5-13acf52493e9-kube-api-access-49vbm\") pod \"payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc\" (UID: \"a8aade61-f68c-4205-bed5-13acf52493e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" Sep 06 14:16:17.264876 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.264855 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8aade61-f68c-4205-bed5-13acf52493e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc\" (UID: \"a8aade61-f68c-4205-bed5-13acf52493e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" Sep 06 14:16:17.271515 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.271492 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49vbm\" (UniqueName: \"kubernetes.io/projected/a8aade61-f68c-4205-bed5-13acf52493e9-kube-api-access-49vbm\") pod \"payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc\" (UID: \"a8aade61-f68c-4205-bed5-13acf52493e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" Sep 06 14:16:17.387846 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.387761 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" Sep 06 14:16:17.660829 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.660796 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc"] Sep 06 14:16:17.662408 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:16:17.662364 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8aade61_f68c_4205_bed5_13acf52493e9.slice/crio-eba99ff50a80243bd1823dcf0db4f4a78d2bb19e57c15b6ae4dddb82fc59bf8c WatchSource:0}: Error finding container eba99ff50a80243bd1823dcf0db4f4a78d2bb19e57c15b6ae4dddb82fc59bf8c: Status 404 returned error can't find the container with id eba99ff50a80243bd1823dcf0db4f4a78d2bb19e57c15b6ae4dddb82fc59bf8c Sep 06 14:16:17.976455 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.976371 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" event={"ID":"a8aade61-f68c-4205-bed5-13acf52493e9","Type":"ContainerStarted","Data":"480911ea9647de2e4ab0216a0ee242070ff9e79aaa4315c878e3b4246a3d7b25"} Sep 06 14:16:17.976455 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.976437 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" Sep 06 14:16:17.976633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.976461 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" event={"ID":"a8aade61-f68c-4205-bed5-13acf52493e9","Type":"ContainerStarted","Data":"eba99ff50a80243bd1823dcf0db4f4a78d2bb19e57c15b6ae4dddb82fc59bf8c"} Sep 06 14:16:17.992731 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:17.992681 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" podStartSLOduration=0.992663795 podStartE2EDuration="992.663795ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:16:17.991378398 +0000 UTC m=+1109.446867277" watchObservedRunningTime="2026-09-06 14:16:17.992663795 +0000 UTC m=+1109.448152690" Sep 06 14:16:23.295487 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:23.295452 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:16:28.810236 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:28.810204 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:28.984439 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:28.984407 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" Sep 06 14:16:34.991935 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:34.991891 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:16:40.190590 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:40.190547 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:47.493787 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:47.493743 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:16:54.368563 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.368527 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:54.699378 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.699298 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr"] Sep 06 14:16:54.703800 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.703774 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.706253 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.706227 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-25ae160a-openshift-default-dockercfg-xhgh6\"" Sep 06 14:16:54.713489 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.713464 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr"] Sep 06 14:16:54.714782 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.714756 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwcwt\" (UniqueName: \"kubernetes.io/projected/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-kube-api-access-gwcwt\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.714951 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.714787 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-credential-socket\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.714951 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.714827 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-workload-certs\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.715097 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.714942 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-podinfo\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.715097 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.714983 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-token\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.715097 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.715009 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-envoy\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.715253 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.715124 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-data\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.715253 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.715174 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-workload-socket\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.715253 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.715191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istiod-ca-cert\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.725649 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.725620 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:54.816575 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.816542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-workload-certs\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.816735 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.816587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-podinfo\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.816783 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.816765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-token\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.816851 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.816832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-envoy\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.816964 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.816947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-data\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.817012 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.816998 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-workload-certs\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.817071 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.817004 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-workload-socket\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.817071 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.817052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istiod-ca-cert\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.817172 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.817118 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwcwt\" (UniqueName: \"kubernetes.io/projected/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-kube-api-access-gwcwt\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.817172 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.817148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-credential-socket\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.817269 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.817230 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-workload-socket\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.817471 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.817427 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-data\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.817471 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.817454 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-credential-socket\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.818027 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.818005 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istiod-ca-cert\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.819347 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.819320 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-envoy\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.819541 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.819521 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-podinfo\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.825013 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.824991 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwcwt\" (UniqueName: \"kubernetes.io/projected/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-kube-api-access-gwcwt\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:54.825878 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:54.825860 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-token\") pod \"e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:55.016866 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:55.016783 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:55.219249 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:55.219215 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:55.289257 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:55.289224 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr"] Sep 06 14:16:55.291278 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:16:55.291249 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4bee7b_6329_4680_ac9e_f91a7fe2df33.slice/crio-645d3be04db876c9439c273d6702d397aa89e755ee06028479aebd21f946f40c WatchSource:0}: Error finding container 645d3be04db876c9439c273d6702d397aa89e755ee06028479aebd21f946f40c: Status 404 returned error can't find the container with id 645d3be04db876c9439c273d6702d397aa89e755ee06028479aebd21f946f40c Sep 06 14:16:55.293753 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:55.293719 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:16:55.293831 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:55.293790 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:16:55.293831 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:55.293817 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:16:56.180102 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:56.180068 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" event={"ID":"dd4bee7b-6329-4680-ac9e-f91a7fe2df33","Type":"ContainerStarted","Data":"0c0bdb156ff3f006c6716e621cb657c91131a0d03d895d897b000a4a31f56704"} Sep 06 14:16:56.180497 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:56.180132 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" event={"ID":"dd4bee7b-6329-4680-ac9e-f91a7fe2df33","Type":"ContainerStarted","Data":"645d3be04db876c9439c273d6702d397aa89e755ee06028479aebd21f946f40c"} Sep 06 14:16:56.198524 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:56.198476 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" podStartSLOduration=2.198459854 podStartE2EDuration="2.198459854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:16:56.196875151 +0000 UTC m=+1147.652364014" watchObservedRunningTime="2026-09-06 14:16:56.198459854 +0000 UTC m=+1147.653948715" Sep 06 14:16:57.017973 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:57.017939 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:57.023067 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:57.023043 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:57.185406 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:57.185374 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:57.186247 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:57.186226 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:16:58.041731 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:58.041700 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:58.139491 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:58.139463 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:59.192346 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.192306 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s"] Sep 06 14:16:59.195540 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.195515 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:16:59.197717 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.197694 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-25ae160a-serving-cert\"" Sep 06 14:16:59.206226 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.206205 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s"] Sep 06 14:16:59.232836 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.232803 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:59.265195 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.265171 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/361ffdb7-1dd2-4116-ad83-17030864a800-metrics-certs\") pod \"maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:16:59.265304 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.265227 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/361ffdb7-1dd2-4116-ad83-17030864a800-openshift-service-ca\") pod \"maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:16:59.265357 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.265305 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/361ffdb7-1dd2-4116-ad83-17030864a800-maas-api-tls\") pod \"maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:16:59.265402 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.265373 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h74q9\" (UniqueName: \"kubernetes.io/projected/361ffdb7-1dd2-4116-ad83-17030864a800-kube-api-access-h74q9\") pod \"maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:16:59.334395 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.334346 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:59.366084 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.366056 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/361ffdb7-1dd2-4116-ad83-17030864a800-openshift-service-ca\") pod \"maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:16:59.366219 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.366101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/361ffdb7-1dd2-4116-ad83-17030864a800-maas-api-tls\") pod \"maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:16:59.366219 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.366136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74q9\" (UniqueName: \"kubernetes.io/projected/361ffdb7-1dd2-4116-ad83-17030864a800-kube-api-access-h74q9\") pod \"maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:16:59.366219 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.366212 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/361ffdb7-1dd2-4116-ad83-17030864a800-metrics-certs\") pod \"maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:16:59.366323 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:16:59.366258 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-serving-cert: secret "maas-api-e2e-mt-25ae160a-serving-cert" not found Sep 06 14:16:59.366357 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:16:59.366331 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/361ffdb7-1dd2-4116-ad83-17030864a800-maas-api-tls podName:361ffdb7-1dd2-4116-ad83-17030864a800 nodeName:}" failed. No retries permitted until 2026-09-06 14:16:59.866311413 +0000 UTC m=+1151.321800255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/361ffdb7-1dd2-4116-ad83-17030864a800-maas-api-tls") pod "maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" (UID: "361ffdb7-1dd2-4116-ad83-17030864a800") : secret "maas-api-e2e-mt-25ae160a-serving-cert" not found Sep 06 14:16:59.366727 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.366707 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/361ffdb7-1dd2-4116-ad83-17030864a800-openshift-service-ca\") pod \"maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:16:59.368525 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.368508 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/361ffdb7-1dd2-4116-ad83-17030864a800-metrics-certs\") pod \"maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:16:59.376541 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.376520 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h74q9\" (UniqueName: \"kubernetes.io/projected/361ffdb7-1dd2-4116-ad83-17030864a800-kube-api-access-h74q9\") pod \"maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:16:59.466108 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.466041 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:59.691516 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.691489 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:59.785130 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.785093 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:16:59.871114 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.871070 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/361ffdb7-1dd2-4116-ad83-17030864a800-maas-api-tls\") pod \"maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:16:59.874657 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:16:59.874040 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/361ffdb7-1dd2-4116-ad83-17030864a800-maas-api-tls\") pod \"maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:17:00.108928 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:00.108832 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:17:00.378826 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:00.378801 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s"] Sep 06 14:17:00.381144 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:17:00.381117 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361ffdb7_1dd2_4116_ad83_17030864a800.slice/crio-4f9e4be72c20fe6775b4d3cf43aa345442b41503d24e14e46f19c58637cf6425 WatchSource:0}: Error finding container 4f9e4be72c20fe6775b4d3cf43aa345442b41503d24e14e46f19c58637cf6425: Status 404 returned error can't find the container with id 4f9e4be72c20fe6775b4d3cf43aa345442b41503d24e14e46f19c58637cf6425 Sep 06 14:17:01.212114 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:01.212026 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" event={"ID":"361ffdb7-1dd2-4116-ad83-17030864a800","Type":"ContainerStarted","Data":"7fddc3b561815e54d3dc740fd3262941f95b35d03f740a8c727f8a69afe92fd1"} Sep 06 14:17:01.212114 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:01.212089 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" event={"ID":"361ffdb7-1dd2-4116-ad83-17030864a800","Type":"ContainerStarted","Data":"4f9e4be72c20fe6775b4d3cf43aa345442b41503d24e14e46f19c58637cf6425"} Sep 06 14:17:01.212301 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:01.212121 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:17:01.228715 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:01.228662 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" podStartSLOduration=1.7083602230000001 podStartE2EDuration="2.22864444s" totalImagesPullingTime="520.284217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:16:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:17:01.226774311 +0000 UTC m=+1152.682263175" watchObservedRunningTime="2026-09-06 14:17:01.22864444 +0000 UTC m=+1152.684133303" Sep 06 14:17:04.392418 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:04.392383 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:17:07.228707 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:07.227962 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:17:22.953745 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:22.953663 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:17:23.595033 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:23.594995 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:17:28.226090 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:28.226053 2577 patch_prober.go:28] interesting pod/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 06 14:17:28.226591 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:28.226127 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" podUID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 06 14:17:29.401387 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:29.401293 2577 generic.go:356] "Generic (PLEG): container finished" podID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" containerID="0c0bdb156ff3f006c6716e621cb657c91131a0d03d895d897b000a4a31f56704" exitCode=0 Sep 06 14:17:29.401387 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:29.401355 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" event={"ID":"dd4bee7b-6329-4680-ac9e-f91a7fe2df33","Type":"ContainerDied","Data":"0c0bdb156ff3f006c6716e621cb657c91131a0d03d895d897b000a4a31f56704"} Sep 06 14:17:29.401811 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:29.401697 2577 scope.go:117] "RemoveContainer" containerID="0c0bdb156ff3f006c6716e621cb657c91131a0d03d895d897b000a4a31f56704" Sep 06 14:17:29.402742 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:29.402713 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:17:29.402793 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:29.402779 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:17:29.402837 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:29.402814 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:17:30.410027 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:30.409991 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" event={"ID":"dd4bee7b-6329-4680-ac9e-f91a7fe2df33","Type":"ContainerStarted","Data":"712e8f2136dcd70232df0988e1db88a2609c964ba7ea3400909659c0c3b98a56"} Sep 06 14:17:30.647409 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:30.647373 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:17:31.017896 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:31.017859 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:17:31.018110 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:31.018076 2577 patch_prober.go:28] interesting pod/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 06 14:17:31.018191 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:31.018133 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" podUID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 06 14:17:32.017694 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:32.017660 2577 patch_prober.go:28] interesting pod/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 06 14:17:32.018111 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:32.017715 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" podUID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 06 14:17:33.017977 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:33.017940 2577 patch_prober.go:28] interesting pod/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 06 14:17:33.018344 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:33.017998 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" podUID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 06 14:17:34.017354 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:34.017321 2577 patch_prober.go:28] interesting pod/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 06 14:17:34.017550 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:34.017379 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" podUID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 06 14:17:35.018109 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:35.018073 2577 patch_prober.go:28] interesting pod/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 06 14:17:35.018468 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:35.018134 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" podUID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 06 14:17:36.021775 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:36.021746 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:17:36.022156 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:36.022127 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:17:36.022836 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:36.022813 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:17:36.415053 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:36.414972 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:17:41.399323 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:17:41.399287 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:18:02.122349 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:02.122312 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:04.887330 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:04.887288 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:18:17.375376 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.375335 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s"] Sep 06 14:18:17.375821 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.375647 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" podUID="361ffdb7-1dd2-4116-ad83-17030864a800" containerName="maas-api" containerID="cri-o://7fddc3b561815e54d3dc740fd3262941f95b35d03f740a8c727f8a69afe92fd1" gracePeriod=30 Sep 06 14:18:17.393531 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.393487 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:17.656336 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.656306 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:18:17.702982 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.702953 2577 generic.go:356] "Generic (PLEG): container finished" podID="361ffdb7-1dd2-4116-ad83-17030864a800" containerID="7fddc3b561815e54d3dc740fd3262941f95b35d03f740a8c727f8a69afe92fd1" exitCode=0 Sep 06 14:18:17.703153 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.702998 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" Sep 06 14:18:17.703153 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.703033 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" event={"ID":"361ffdb7-1dd2-4116-ad83-17030864a800","Type":"ContainerDied","Data":"7fddc3b561815e54d3dc740fd3262941f95b35d03f740a8c727f8a69afe92fd1"} Sep 06 14:18:17.703153 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.703086 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s" event={"ID":"361ffdb7-1dd2-4116-ad83-17030864a800","Type":"ContainerDied","Data":"4f9e4be72c20fe6775b4d3cf43aa345442b41503d24e14e46f19c58637cf6425"} Sep 06 14:18:17.703153 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.703108 2577 scope.go:117] "RemoveContainer" containerID="7fddc3b561815e54d3dc740fd3262941f95b35d03f740a8c727f8a69afe92fd1" Sep 06 14:18:17.712183 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.712169 2577 scope.go:117] "RemoveContainer" containerID="7fddc3b561815e54d3dc740fd3262941f95b35d03f740a8c727f8a69afe92fd1" Sep 06 14:18:17.712438 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:18:17.712422 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fddc3b561815e54d3dc740fd3262941f95b35d03f740a8c727f8a69afe92fd1\": container with ID starting with 7fddc3b561815e54d3dc740fd3262941f95b35d03f740a8c727f8a69afe92fd1 not found: ID does not exist" containerID="7fddc3b561815e54d3dc740fd3262941f95b35d03f740a8c727f8a69afe92fd1" Sep 06 14:18:17.712477 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.712448 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fddc3b561815e54d3dc740fd3262941f95b35d03f740a8c727f8a69afe92fd1"} err="failed to get container status \"7fddc3b561815e54d3dc740fd3262941f95b35d03f740a8c727f8a69afe92fd1\": rpc error: code = NotFound desc = could not find container \"7fddc3b561815e54d3dc740fd3262941f95b35d03f740a8c727f8a69afe92fd1\": container with ID starting with 7fddc3b561815e54d3dc740fd3262941f95b35d03f740a8c727f8a69afe92fd1 not found: ID does not exist" Sep 06 14:18:17.749044 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.749018 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/361ffdb7-1dd2-4116-ad83-17030864a800-openshift-service-ca\") pod \"361ffdb7-1dd2-4116-ad83-17030864a800\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " Sep 06 14:18:17.749164 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.749107 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/361ffdb7-1dd2-4116-ad83-17030864a800-metrics-certs\") pod \"361ffdb7-1dd2-4116-ad83-17030864a800\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " Sep 06 14:18:17.749216 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.749173 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/361ffdb7-1dd2-4116-ad83-17030864a800-maas-api-tls\") pod \"361ffdb7-1dd2-4116-ad83-17030864a800\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " Sep 06 14:18:17.749267 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.749235 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h74q9\" (UniqueName: \"kubernetes.io/projected/361ffdb7-1dd2-4116-ad83-17030864a800-kube-api-access-h74q9\") pod \"361ffdb7-1dd2-4116-ad83-17030864a800\" (UID: \"361ffdb7-1dd2-4116-ad83-17030864a800\") " Sep 06 14:18:17.749410 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.749384 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361ffdb7-1dd2-4116-ad83-17030864a800-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "361ffdb7-1dd2-4116-ad83-17030864a800" (UID: "361ffdb7-1dd2-4116-ad83-17030864a800"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:18:17.749681 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.749655 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/361ffdb7-1dd2-4116-ad83-17030864a800-openshift-service-ca\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:18:17.751325 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.751300 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/361ffdb7-1dd2-4116-ad83-17030864a800-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "361ffdb7-1dd2-4116-ad83-17030864a800" (UID: "361ffdb7-1dd2-4116-ad83-17030864a800"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:18:17.751673 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.751646 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/361ffdb7-1dd2-4116-ad83-17030864a800-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "361ffdb7-1dd2-4116-ad83-17030864a800" (UID: "361ffdb7-1dd2-4116-ad83-17030864a800"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:18:17.751757 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.751687 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361ffdb7-1dd2-4116-ad83-17030864a800-kube-api-access-h74q9" (OuterVolumeSpecName: "kube-api-access-h74q9") pod "361ffdb7-1dd2-4116-ad83-17030864a800" (UID: "361ffdb7-1dd2-4116-ad83-17030864a800"). InnerVolumeSpecName "kube-api-access-h74q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:18:17.850635 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.850604 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/361ffdb7-1dd2-4116-ad83-17030864a800-metrics-certs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:18:17.850635 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.850631 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/361ffdb7-1dd2-4116-ad83-17030864a800-maas-api-tls\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:18:17.850795 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:17.850644 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h74q9\" (UniqueName: \"kubernetes.io/projected/361ffdb7-1dd2-4116-ad83-17030864a800-kube-api-access-h74q9\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:18:18.024748 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:18.024713 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s"] Sep 06 14:18:18.030687 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:18.030662 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25ae160a-5c65476f84-fvd8s"] Sep 06 14:18:18.587837 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:18.587802 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:19.068599 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:19.068570 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="361ffdb7-1dd2-4116-ad83-17030864a800" path="/var/lib/kubelet/pods/361ffdb7-1dd2-4116-ad83-17030864a800/volumes" Sep 06 14:18:19.386550 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:19.386464 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:18:20.576225 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:20.576191 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:23.692280 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:23.692218 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:23.739674 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:23.739646 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:25.600539 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:25.600508 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:33.186759 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:33.186725 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:18:40.233464 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:40.233417 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:41.415381 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:41.415302 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:44.102047 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:44.102012 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:44.816321 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:44.816284 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:45.198877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:45.198799 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:18:49.008536 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:49.008495 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:49.246162 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:49.246128 2577 scope.go:117] "RemoveContainer" containerID="4da7e1c6fd4a83cecb0a0edd8b8c9328d383521b763f77f4105022d6c9a28a54" Sep 06 14:18:50.457538 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:50.457492 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:55.987473 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:55.987441 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:56.088097 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:56.088065 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:56.211121 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:56.211087 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr"] Sep 06 14:18:56.211381 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:56.211351 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" podUID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" containerName="istio-proxy" containerID="cri-o://712e8f2136dcd70232df0988e1db88a2609c964ba7ea3400909659c0c3b98a56" gracePeriod=30 Sep 06 14:18:56.238737 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:56.238662 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:57.429643 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:57.429611 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:57.529931 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:57.529876 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:57.630065 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:57.630038 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:58.185672 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:58.185636 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:58.288830 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:58.288796 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:58.387437 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:58.387397 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:18:59.602451 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:18:59.602420 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:01.471061 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.471038 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:19:01.562272 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.562245 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwcwt\" (UniqueName: \"kubernetes.io/projected/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-kube-api-access-gwcwt\") pod \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " Sep 06 14:19:01.562450 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.562301 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-workload-certs\") pod \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " Sep 06 14:19:01.562450 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.562319 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-data\") pod \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " Sep 06 14:19:01.562450 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.562337 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-credential-socket\") pod \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " Sep 06 14:19:01.562450 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.562356 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istiod-ca-cert\") pod \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " Sep 06 14:19:01.562450 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.562400 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-token\") pod \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " Sep 06 14:19:01.562450 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.562436 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-podinfo\") pod \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " Sep 06 14:19:01.562773 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.562494 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-workload-socket\") pod \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " Sep 06 14:19:01.562773 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.562560 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dd4bee7b-6329-4680-ac9e-f91a7fe2df33" (UID: "dd4bee7b-6329-4680-ac9e-f91a7fe2df33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:19:01.562773 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.562600 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-data" (OuterVolumeSpecName: "istio-data") pod "dd4bee7b-6329-4680-ac9e-f91a7fe2df33" (UID: "dd4bee7b-6329-4680-ac9e-f91a7fe2df33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:19:01.562773 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.562647 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dd4bee7b-6329-4680-ac9e-f91a7fe2df33" (UID: "dd4bee7b-6329-4680-ac9e-f91a7fe2df33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:19:01.562773 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.562601 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-envoy\") pod \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\" (UID: \"dd4bee7b-6329-4680-ac9e-f91a7fe2df33\") " Sep 06 14:19:01.563125 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.562838 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dd4bee7b-6329-4680-ac9e-f91a7fe2df33" (UID: "dd4bee7b-6329-4680-ac9e-f91a7fe2df33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:19:01.563125 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.562887 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dd4bee7b-6329-4680-ac9e-f91a7fe2df33" (UID: "dd4bee7b-6329-4680-ac9e-f91a7fe2df33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:19:01.563252 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.563182 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-workload-certs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:19:01.563252 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.563202 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-data\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:19:01.563252 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.563214 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-credential-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:19:01.563252 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.563227 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istiod-ca-cert\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:19:01.563252 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.563241 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-workload-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:19:01.564537 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.564514 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-kube-api-access-gwcwt" (OuterVolumeSpecName: "kube-api-access-gwcwt") pod "dd4bee7b-6329-4680-ac9e-f91a7fe2df33" (UID: "dd4bee7b-6329-4680-ac9e-f91a7fe2df33"). InnerVolumeSpecName "kube-api-access-gwcwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:19:01.564718 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.564697 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dd4bee7b-6329-4680-ac9e-f91a7fe2df33" (UID: "dd4bee7b-6329-4680-ac9e-f91a7fe2df33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 14:19:01.564780 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.564705 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-token" (OuterVolumeSpecName: "istio-token") pod "dd4bee7b-6329-4680-ac9e-f91a7fe2df33" (UID: "dd4bee7b-6329-4680-ac9e-f91a7fe2df33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:19:01.564780 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.564716 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dd4bee7b-6329-4680-ac9e-f91a7fe2df33" (UID: "dd4bee7b-6329-4680-ac9e-f91a7fe2df33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:19:01.664056 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.664016 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-envoy\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:19:01.664056 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.664048 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwcwt\" (UniqueName: \"kubernetes.io/projected/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-kube-api-access-gwcwt\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:19:01.664056 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.664061 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-token\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:19:01.664290 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.664073 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4bee7b-6329-4680-ac9e-f91a7fe2df33-istio-podinfo\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:19:01.952358 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.952269 2577 generic.go:356] "Generic (PLEG): container finished" podID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" containerID="712e8f2136dcd70232df0988e1db88a2609c964ba7ea3400909659c0c3b98a56" exitCode=0 Sep 06 14:19:01.952358 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.952317 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" Sep 06 14:19:01.952358 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.952335 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" event={"ID":"dd4bee7b-6329-4680-ac9e-f91a7fe2df33","Type":"ContainerDied","Data":"712e8f2136dcd70232df0988e1db88a2609c964ba7ea3400909659c0c3b98a56"} Sep 06 14:19:01.952615 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.952389 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr" event={"ID":"dd4bee7b-6329-4680-ac9e-f91a7fe2df33","Type":"ContainerDied","Data":"645d3be04db876c9439c273d6702d397aa89e755ee06028479aebd21f946f40c"} Sep 06 14:19:01.952615 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.952419 2577 scope.go:117] "RemoveContainer" containerID="712e8f2136dcd70232df0988e1db88a2609c964ba7ea3400909659c0c3b98a56" Sep 06 14:19:01.961899 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.961880 2577 scope.go:117] "RemoveContainer" containerID="0c0bdb156ff3f006c6716e621cb657c91131a0d03d895d897b000a4a31f56704" Sep 06 14:19:01.970287 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.970262 2577 scope.go:117] "RemoveContainer" containerID="712e8f2136dcd70232df0988e1db88a2609c964ba7ea3400909659c0c3b98a56" Sep 06 14:19:01.970608 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:19:01.970589 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712e8f2136dcd70232df0988e1db88a2609c964ba7ea3400909659c0c3b98a56\": container with ID starting with 712e8f2136dcd70232df0988e1db88a2609c964ba7ea3400909659c0c3b98a56 not found: ID does not exist" containerID="712e8f2136dcd70232df0988e1db88a2609c964ba7ea3400909659c0c3b98a56" Sep 06 14:19:01.970675 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.970616 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712e8f2136dcd70232df0988e1db88a2609c964ba7ea3400909659c0c3b98a56"} err="failed to get container status \"712e8f2136dcd70232df0988e1db88a2609c964ba7ea3400909659c0c3b98a56\": rpc error: code = NotFound desc = could not find container \"712e8f2136dcd70232df0988e1db88a2609c964ba7ea3400909659c0c3b98a56\": container with ID starting with 712e8f2136dcd70232df0988e1db88a2609c964ba7ea3400909659c0c3b98a56 not found: ID does not exist" Sep 06 14:19:01.970675 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.970633 2577 scope.go:117] "RemoveContainer" containerID="0c0bdb156ff3f006c6716e621cb657c91131a0d03d895d897b000a4a31f56704" Sep 06 14:19:01.970900 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:19:01.970867 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0bdb156ff3f006c6716e621cb657c91131a0d03d895d897b000a4a31f56704\": container with ID starting with 0c0bdb156ff3f006c6716e621cb657c91131a0d03d895d897b000a4a31f56704 not found: ID does not exist" containerID="0c0bdb156ff3f006c6716e621cb657c91131a0d03d895d897b000a4a31f56704" Sep 06 14:19:01.971016 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.970894 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0bdb156ff3f006c6716e621cb657c91131a0d03d895d897b000a4a31f56704"} err="failed to get container status \"0c0bdb156ff3f006c6716e621cb657c91131a0d03d895d897b000a4a31f56704\": rpc error: code = NotFound desc = could not find container \"0c0bdb156ff3f006c6716e621cb657c91131a0d03d895d897b000a4a31f56704\": container with ID starting with 0c0bdb156ff3f006c6716e621cb657c91131a0d03d895d897b000a4a31f56704 not found: ID does not exist" Sep 06 14:19:01.972738 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.972719 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr"] Sep 06 14:19:01.978748 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:01.978728 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-25ae160a-openshift-default-7f48c49fb8-pgzgr"] Sep 06 14:19:03.068567 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:03.068522 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" path="/var/lib/kubelet/pods/dd4bee7b-6329-4680-ac9e-f91a7fe2df33/volumes" Sep 06 14:19:04.286051 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:04.286017 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:19:05.396025 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:05.395995 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:05.459853 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:05.459824 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:06.045450 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:06.045416 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:11.752497 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:11.752466 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:11.811012 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:11.810978 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:11.880832 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:11.880804 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:11.981204 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:11.981177 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:15.821112 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:15.821076 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:15.869238 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:15.869215 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:18.587193 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:18.587158 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:19:20.015989 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:20.015959 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:22.413558 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:22.413522 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:22.624696 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:22.624665 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:34.787537 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:34.787504 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:19:43.515553 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:43.515514 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:49.257537 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.257499 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:49.357466 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.357437 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:49.639790 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.639760 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:49.871785 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.871745 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn"] Sep 06 14:19:49.872293 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.872277 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361ffdb7-1dd2-4116-ad83-17030864a800" containerName="maas-api" Sep 06 14:19:49.872293 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.872294 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="361ffdb7-1dd2-4116-ad83-17030864a800" containerName="maas-api" Sep 06 14:19:49.872453 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.872306 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" containerName="istio-proxy" Sep 06 14:19:49.872453 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.872311 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" containerName="istio-proxy" Sep 06 14:19:49.872453 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.872322 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" containerName="istio-proxy" Sep 06 14:19:49.872453 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.872328 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" containerName="istio-proxy" Sep 06 14:19:49.872453 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.872428 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" containerName="istio-proxy" Sep 06 14:19:49.872453 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.872439 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="361ffdb7-1dd2-4116-ad83-17030864a800" containerName="maas-api" Sep 06 14:19:49.875539 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.875518 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:49.878156 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.878130 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-61d831-openshift-default-dockercfg-52lnf\"" Sep 06 14:19:49.886078 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.886057 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn"] Sep 06 14:19:49.901757 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:49.901678 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:50.010618 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.010578 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/480b8385-d86f-4f85-a48f-b85de93a36c1-istiod-ca-cert\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.010789 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.010627 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-token\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.010789 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.010737 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-workload-certs\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.010789 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.010772 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-credential-socket\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.010991 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.010836 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-workload-socket\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.010991 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.010891 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-envoy\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.011174 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.010990 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnr8\" (UniqueName: \"kubernetes.io/projected/480b8385-d86f-4f85-a48f-b85de93a36c1-kube-api-access-zwnr8\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.011174 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.011035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-data\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.011174 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.011061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-podinfo\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.017733 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.017712 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:50.112542 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.112507 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-podinfo\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.112734 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.112555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/480b8385-d86f-4f85-a48f-b85de93a36c1-istiod-ca-cert\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.112734 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.112580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-token\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.112734 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.112633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-workload-certs\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.112734 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.112653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-credential-socket\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.112734 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.112688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-workload-socket\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.112734 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.112725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-envoy\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.113107 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.112772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnr8\" (UniqueName: \"kubernetes.io/projected/480b8385-d86f-4f85-a48f-b85de93a36c1-kube-api-access-zwnr8\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.113107 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.112805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-data\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.113222 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.113144 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-workload-socket\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.113222 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.113165 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-data\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.113330 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.113225 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-credential-socket\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.113387 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.113352 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-workload-certs\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.113387 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.113361 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/480b8385-d86f-4f85-a48f-b85de93a36c1-istiod-ca-cert\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.115033 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.115009 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-envoy\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.115138 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.115121 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-podinfo\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.119717 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.119694 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-token\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.120026 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.120009 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnr8\" (UniqueName: \"kubernetes.io/projected/480b8385-d86f-4f85-a48f-b85de93a36c1-kube-api-access-zwnr8\") pod \"e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.186900 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.186829 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:50.458170 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.458141 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn"] Sep 06 14:19:50.460340 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:19:50.460310 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480b8385_d86f_4f85_a48f_b85de93a36c1.slice/crio-aefecee11d6d5011249ea220acb3ee9db8c520deeaccfb46c2dbf59fafb3c8f7 WatchSource:0}: Error finding container aefecee11d6d5011249ea220acb3ee9db8c520deeaccfb46c2dbf59fafb3c8f7: Status 404 returned error can't find the container with id aefecee11d6d5011249ea220acb3ee9db8c520deeaccfb46c2dbf59fafb3c8f7 Sep 06 14:19:50.462583 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.462548 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:19:50.462661 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.462614 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:19:50.462661 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:50.462642 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:19:51.219861 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:51.219826 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" event={"ID":"480b8385-d86f-4f85-a48f-b85de93a36c1","Type":"ContainerStarted","Data":"c868992b34d7377e531fa0ea1c5cc6656d2fd0c13b3c6d180e6ba42621f2eccf"} Sep 06 14:19:51.220060 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:51.219890 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" event={"ID":"480b8385-d86f-4f85-a48f-b85de93a36c1","Type":"ContainerStarted","Data":"aefecee11d6d5011249ea220acb3ee9db8c520deeaccfb46c2dbf59fafb3c8f7"} Sep 06 14:19:51.239024 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:51.238972 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" podStartSLOduration=2.238957552 podStartE2EDuration="2.238957552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:19:51.236928364 +0000 UTC m=+1322.692417223" watchObservedRunningTime="2026-09-06 14:19:51.238957552 +0000 UTC m=+1322.694446413" Sep 06 14:19:52.187789 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:52.187756 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:52.193082 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:52.193057 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:52.225435 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:52.225404 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:52.226700 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:52.226679 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:19:53.100988 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:53.100953 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:53.193419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:53.193366 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:54.098445 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.098410 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:19:54.213293 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.213265 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz"] Sep 06 14:19:54.213787 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.213773 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd4bee7b-6329-4680-ac9e-f91a7fe2df33" containerName="istio-proxy" Sep 06 14:19:54.216227 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.216203 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:19:54.218394 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.218376 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-61d831-serving-cert\"" Sep 06 14:19:54.223586 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.223564 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz"] Sep 06 14:19:54.240708 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.240681 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:54.342469 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.342437 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:54.352627 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.352561 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d803825f-8585-4db4-b705-c22acaf29a3c-openshift-service-ca\") pod \"maas-api-e2e-api-a-61d831-5846556d4c-k6tjz\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:19:54.352627 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.352614 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d803825f-8585-4db4-b705-c22acaf29a3c-maas-api-tls\") pod \"maas-api-e2e-api-a-61d831-5846556d4c-k6tjz\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:19:54.352791 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.352723 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnhhv\" (UniqueName: \"kubernetes.io/projected/d803825f-8585-4db4-b705-c22acaf29a3c-kube-api-access-vnhhv\") pod \"maas-api-e2e-api-a-61d831-5846556d4c-k6tjz\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:19:54.352791 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.352752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d803825f-8585-4db4-b705-c22acaf29a3c-metrics-certs\") pod \"maas-api-e2e-api-a-61d831-5846556d4c-k6tjz\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:19:54.453362 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.453334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d803825f-8585-4db4-b705-c22acaf29a3c-openshift-service-ca\") pod \"maas-api-e2e-api-a-61d831-5846556d4c-k6tjz\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:19:54.453538 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.453383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d803825f-8585-4db4-b705-c22acaf29a3c-maas-api-tls\") pod \"maas-api-e2e-api-a-61d831-5846556d4c-k6tjz\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:19:54.453538 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.453434 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnhhv\" (UniqueName: \"kubernetes.io/projected/d803825f-8585-4db4-b705-c22acaf29a3c-kube-api-access-vnhhv\") pod \"maas-api-e2e-api-a-61d831-5846556d4c-k6tjz\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:19:54.453538 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.453462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d803825f-8585-4db4-b705-c22acaf29a3c-metrics-certs\") pod \"maas-api-e2e-api-a-61d831-5846556d4c-k6tjz\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:19:54.454036 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.454013 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d803825f-8585-4db4-b705-c22acaf29a3c-openshift-service-ca\") pod \"maas-api-e2e-api-a-61d831-5846556d4c-k6tjz\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:19:54.455790 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.455769 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d803825f-8585-4db4-b705-c22acaf29a3c-maas-api-tls\") pod \"maas-api-e2e-api-a-61d831-5846556d4c-k6tjz\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:19:54.455885 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.455816 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d803825f-8585-4db4-b705-c22acaf29a3c-metrics-certs\") pod \"maas-api-e2e-api-a-61d831-5846556d4c-k6tjz\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:19:54.460676 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.460657 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnhhv\" (UniqueName: \"kubernetes.io/projected/d803825f-8585-4db4-b705-c22acaf29a3c-kube-api-access-vnhhv\") pod \"maas-api-e2e-api-a-61d831-5846556d4c-k6tjz\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:19:54.482881 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.482853 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:54.530347 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.530323 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:19:54.803726 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.803698 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz"] Sep 06 14:19:54.805613 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:19:54.805586 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd803825f_8585_4db4_b705_c22acaf29a3c.slice/crio-5ae29d2c30c33cf38c2eb7f4aebce502c498b45c1437f6698268a1e41c0aa6cc WatchSource:0}: Error finding container 5ae29d2c30c33cf38c2eb7f4aebce502c498b45c1437f6698268a1e41c0aa6cc: Status 404 returned error can't find the container with id 5ae29d2c30c33cf38c2eb7f4aebce502c498b45c1437f6698268a1e41c0aa6cc Sep 06 14:19:54.816940 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.816917 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:54.915079 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:54.915047 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:19:55.244729 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:55.244705 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" event={"ID":"d803825f-8585-4db4-b705-c22acaf29a3c","Type":"ContainerStarted","Data":"5ae29d2c30c33cf38c2eb7f4aebce502c498b45c1437f6698268a1e41c0aa6cc"} Sep 06 14:19:56.252745 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:56.252714 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" event={"ID":"d803825f-8585-4db4-b705-c22acaf29a3c","Type":"ContainerStarted","Data":"23b47f0b728a31a0ba6fb7c9e1516fee173c27741b4070b4b511601e3028ba08"} Sep 06 14:19:56.253139 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:56.252786 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:19:56.268218 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:19:56.268170 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" podStartSLOduration=1.8464125139999998 podStartE2EDuration="2.268155286s" totalImagesPullingTime="421.742772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:19:56.266238219 +0000 UTC m=+1327.721727084" watchObservedRunningTime="2026-09-06 14:19:56.268155286 +0000 UTC m=+1327.723644145" Sep 06 14:20:02.263734 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:02.263704 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:20:11.686610 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:11.686514 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:20:14.362897 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:14.362866 2577 generic.go:356] "Generic (PLEG): container finished" podID="480b8385-d86f-4f85-a48f-b85de93a36c1" containerID="c868992b34d7377e531fa0ea1c5cc6656d2fd0c13b3c6d180e6ba42621f2eccf" exitCode=0 Sep 06 14:20:14.363336 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:14.362946 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" event={"ID":"480b8385-d86f-4f85-a48f-b85de93a36c1","Type":"ContainerDied","Data":"c868992b34d7377e531fa0ea1c5cc6656d2fd0c13b3c6d180e6ba42621f2eccf"} Sep 06 14:20:14.363381 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:14.363344 2577 scope.go:117] "RemoveContainer" containerID="c868992b34d7377e531fa0ea1c5cc6656d2fd0c13b3c6d180e6ba42621f2eccf" Sep 06 14:20:14.364457 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:14.364432 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:20:14.364533 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:14.364497 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:20:14.364533 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:14.364528 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:20:14.455180 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:14.455135 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:15.370503 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:15.370464 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" event={"ID":"480b8385-d86f-4f85-a48f-b85de93a36c1","Type":"ContainerStarted","Data":"7d324f274b32b3347dd60a4f5b64045953e878fbbbda54c2e40d574215fb515b"} Sep 06 14:20:15.603169 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:15.603135 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:16.187301 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:16.187262 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:20:16.187501 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:16.187483 2577 patch_prober.go:28] interesting pod/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 06 14:20:16.187550 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:16.187537 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" podUID="480b8385-d86f-4f85-a48f-b85de93a36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 06 14:20:17.187364 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:17.187332 2577 patch_prober.go:28] interesting pod/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 06 14:20:17.187759 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:17.187395 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" podUID="480b8385-d86f-4f85-a48f-b85de93a36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 06 14:20:18.005498 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:18.005457 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:18.187606 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:18.187571 2577 patch_prober.go:28] interesting pod/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 06 14:20:18.188045 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:18.187636 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" podUID="480b8385-d86f-4f85-a48f-b85de93a36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 06 14:20:18.419245 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:18.419211 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:18.711011 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:18.710929 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:18.809750 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:18.809713 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:19.188201 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:19.188162 2577 patch_prober.go:28] interesting pod/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 06 14:20:19.188571 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:19.188222 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" podUID="480b8385-d86f-4f85-a48f-b85de93a36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 06 14:20:20.187265 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:20.187223 2577 patch_prober.go:28] interesting pod/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 06 14:20:20.187565 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:20.187284 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" podUID="480b8385-d86f-4f85-a48f-b85de93a36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 06 14:20:20.927395 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:20.927353 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:21.191044 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:21.190972 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:20:21.191242 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:21.191219 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:20:21.191967 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:21.191948 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:20:21.549007 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:21.548973 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:21.647114 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:21.647058 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:22.724518 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:22.724471 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:22.822101 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:22.822069 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:22.932500 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:22.932471 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:23.268587 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:23.268554 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:23.370488 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:23.370459 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:23.401340 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:23.401309 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf"] Sep 06 14:20:23.405356 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:23.405337 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" Sep 06 14:20:23.407606 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:23.407584 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9dd616\"" Sep 06 14:20:23.407772 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:23.407751 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9dd616-dockercfg-q9wdl\"" Sep 06 14:20:23.415585 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:23.415562 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf"] Sep 06 14:20:23.532654 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:23.532579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgz9n\" (UniqueName: \"kubernetes.io/projected/07caa2e4-73bd-4a30-87b9-9cc23c08df84-kube-api-access-hgz9n\") pod \"payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf\" (UID: \"07caa2e4-73bd-4a30-87b9-9cc23c08df84\") " pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" Sep 06 14:20:23.532654 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:23.532636 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07caa2e4-73bd-4a30-87b9-9cc23c08df84-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf\" (UID: \"07caa2e4-73bd-4a30-87b9-9cc23c08df84\") " pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" Sep 06 14:20:23.633493 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:23.633462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgz9n\" (UniqueName: \"kubernetes.io/projected/07caa2e4-73bd-4a30-87b9-9cc23c08df84-kube-api-access-hgz9n\") pod \"payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf\" (UID: \"07caa2e4-73bd-4a30-87b9-9cc23c08df84\") " pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" Sep 06 14:20:23.633642 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:23.633515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07caa2e4-73bd-4a30-87b9-9cc23c08df84-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf\" (UID: \"07caa2e4-73bd-4a30-87b9-9cc23c08df84\") " pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" Sep 06 14:20:23.634177 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:23.634158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07caa2e4-73bd-4a30-87b9-9cc23c08df84-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf\" (UID: \"07caa2e4-73bd-4a30-87b9-9cc23c08df84\") " pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" Sep 06 14:20:23.641148 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:23.641130 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgz9n\" (UniqueName: \"kubernetes.io/projected/07caa2e4-73bd-4a30-87b9-9cc23c08df84-kube-api-access-hgz9n\") pod \"payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf\" (UID: \"07caa2e4-73bd-4a30-87b9-9cc23c08df84\") " pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" Sep 06 14:20:23.718185 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:23.718152 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" Sep 06 14:20:23.989205 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:23.989177 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf"] Sep 06 14:20:23.990399 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:20:23.990374 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07caa2e4_73bd_4a30_87b9_9cc23c08df84.slice/crio-fb7d6c62c53b660e3a3c90cf22d8b61d8b0c59a064dc8bd748d5500526fae787 WatchSource:0}: Error finding container fb7d6c62c53b660e3a3c90cf22d8b61d8b0c59a064dc8bd748d5500526fae787: Status 404 returned error can't find the container with id fb7d6c62c53b660e3a3c90cf22d8b61d8b0c59a064dc8bd748d5500526fae787 Sep 06 14:20:24.426966 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:24.426930 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" event={"ID":"07caa2e4-73bd-4a30-87b9-9cc23c08df84","Type":"ContainerStarted","Data":"c9cd87dc493e2861513cce673c4d84a01f48910cfe4d14c31e75621e0824e14e"} Sep 06 14:20:24.427164 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:24.426982 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" event={"ID":"07caa2e4-73bd-4a30-87b9-9cc23c08df84","Type":"ContainerStarted","Data":"fb7d6c62c53b660e3a3c90cf22d8b61d8b0c59a064dc8bd748d5500526fae787"} Sep 06 14:20:24.427164 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:24.427038 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" Sep 06 14:20:24.443447 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:24.443405 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" podStartSLOduration=1.443393121 podStartE2EDuration="1.443393121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:20:24.442049225 +0000 UTC m=+1355.897538089" watchObservedRunningTime="2026-09-06 14:20:24.443393121 +0000 UTC m=+1355.898881984" Sep 06 14:20:29.290691 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:29.290655 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:20:35.434550 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:35.434522 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" Sep 06 14:20:35.820513 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:35.820480 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:41.692503 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:41.692464 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:20:46.471140 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:46.471105 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:48.720201 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:48.720168 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:20:49.295046 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:49.295018 2577 scope.go:117] "RemoveContainer" containerID="67a35d5a6e99cd8450a4e27664caec2c4843dcde10fd1a56736a5af749b8d56e" Sep 06 14:20:49.320971 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:20:49.320947 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:04.586157 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.586121 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz"] Sep 06 14:21:04.586683 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.586385 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" podUID="d803825f-8585-4db4-b705-c22acaf29a3c" containerName="maas-api" containerID="cri-o://23b47f0b728a31a0ba6fb7c9e1516fee173c27741b4070b4b511601e3028ba08" gracePeriod=30 Sep 06 14:21:04.613098 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.613072 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:04.713303 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.713242 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:04.783246 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.783203 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:04.833589 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.833563 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:04.835712 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.835696 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:21:04.909000 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.908900 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d803825f-8585-4db4-b705-c22acaf29a3c-maas-api-tls\") pod \"d803825f-8585-4db4-b705-c22acaf29a3c\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " Sep 06 14:21:04.909000 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.908951 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d803825f-8585-4db4-b705-c22acaf29a3c-openshift-service-ca\") pod \"d803825f-8585-4db4-b705-c22acaf29a3c\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " Sep 06 14:21:04.909198 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.909043 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnhhv\" (UniqueName: \"kubernetes.io/projected/d803825f-8585-4db4-b705-c22acaf29a3c-kube-api-access-vnhhv\") pod \"d803825f-8585-4db4-b705-c22acaf29a3c\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " Sep 06 14:21:04.909198 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.909103 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d803825f-8585-4db4-b705-c22acaf29a3c-metrics-certs\") pod \"d803825f-8585-4db4-b705-c22acaf29a3c\" (UID: \"d803825f-8585-4db4-b705-c22acaf29a3c\") " Sep 06 14:21:04.909359 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.909332 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d803825f-8585-4db4-b705-c22acaf29a3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d803825f-8585-4db4-b705-c22acaf29a3c" (UID: "d803825f-8585-4db4-b705-c22acaf29a3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:21:04.909557 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.909539 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d803825f-8585-4db4-b705-c22acaf29a3c-openshift-service-ca\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:04.911140 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.911112 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d803825f-8585-4db4-b705-c22acaf29a3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d803825f-8585-4db4-b705-c22acaf29a3c" (UID: "d803825f-8585-4db4-b705-c22acaf29a3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:21:04.911250 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.911148 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d803825f-8585-4db4-b705-c22acaf29a3c-kube-api-access-vnhhv" (OuterVolumeSpecName: "kube-api-access-vnhhv") pod "d803825f-8585-4db4-b705-c22acaf29a3c" (UID: "d803825f-8585-4db4-b705-c22acaf29a3c"). InnerVolumeSpecName "kube-api-access-vnhhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:21:04.911250 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:04.911191 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d803825f-8585-4db4-b705-c22acaf29a3c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d803825f-8585-4db4-b705-c22acaf29a3c" (UID: "d803825f-8585-4db4-b705-c22acaf29a3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:21:05.010750 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:05.010715 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnhhv\" (UniqueName: \"kubernetes.io/projected/d803825f-8585-4db4-b705-c22acaf29a3c-kube-api-access-vnhhv\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:05.010750 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:05.010745 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d803825f-8585-4db4-b705-c22acaf29a3c-metrics-certs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:05.010950 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:05.010761 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d803825f-8585-4db4-b705-c22acaf29a3c-maas-api-tls\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:05.136842 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:05.136809 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:05.654702 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:05.654667 2577 generic.go:356] "Generic (PLEG): container finished" podID="d803825f-8585-4db4-b705-c22acaf29a3c" containerID="23b47f0b728a31a0ba6fb7c9e1516fee173c27741b4070b4b511601e3028ba08" exitCode=0 Sep 06 14:21:05.655116 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:05.654717 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" Sep 06 14:21:05.655116 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:05.654719 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" event={"ID":"d803825f-8585-4db4-b705-c22acaf29a3c","Type":"ContainerDied","Data":"23b47f0b728a31a0ba6fb7c9e1516fee173c27741b4070b4b511601e3028ba08"} Sep 06 14:21:05.655116 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:05.654772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz" event={"ID":"d803825f-8585-4db4-b705-c22acaf29a3c","Type":"ContainerDied","Data":"5ae29d2c30c33cf38c2eb7f4aebce502c498b45c1437f6698268a1e41c0aa6cc"} Sep 06 14:21:05.655116 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:05.654804 2577 scope.go:117] "RemoveContainer" containerID="23b47f0b728a31a0ba6fb7c9e1516fee173c27741b4070b4b511601e3028ba08" Sep 06 14:21:05.664673 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:05.664655 2577 scope.go:117] "RemoveContainer" containerID="23b47f0b728a31a0ba6fb7c9e1516fee173c27741b4070b4b511601e3028ba08" Sep 06 14:21:05.665015 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:21:05.664994 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b47f0b728a31a0ba6fb7c9e1516fee173c27741b4070b4b511601e3028ba08\": container with ID starting with 23b47f0b728a31a0ba6fb7c9e1516fee173c27741b4070b4b511601e3028ba08 not found: ID does not exist" containerID="23b47f0b728a31a0ba6fb7c9e1516fee173c27741b4070b4b511601e3028ba08" Sep 06 14:21:05.665099 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:05.665027 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b47f0b728a31a0ba6fb7c9e1516fee173c27741b4070b4b511601e3028ba08"} err="failed to get container status \"23b47f0b728a31a0ba6fb7c9e1516fee173c27741b4070b4b511601e3028ba08\": rpc error: code = NotFound desc = could not find container \"23b47f0b728a31a0ba6fb7c9e1516fee173c27741b4070b4b511601e3028ba08\": container with ID starting with 23b47f0b728a31a0ba6fb7c9e1516fee173c27741b4070b4b511601e3028ba08 not found: ID does not exist" Sep 06 14:21:05.670884 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:05.670863 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz"] Sep 06 14:21:05.674574 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:05.674552 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61d831-5846556d4c-k6tjz"] Sep 06 14:21:07.068253 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:07.068217 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d803825f-8585-4db4-b705-c22acaf29a3c" path="/var/lib/kubelet/pods/d803825f-8585-4db4-b705-c22acaf29a3c/volumes" Sep 06 14:21:09.793468 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:09.793431 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:21:10.008589 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:10.008552 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:11.282767 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:11.282733 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:11.341971 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:11.341933 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:11.513580 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:11.513550 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn"] Sep 06 14:21:11.513770 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:11.513747 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" podUID="480b8385-d86f-4f85-a48f-b85de93a36c1" containerName="istio-proxy" containerID="cri-o://7d324f274b32b3347dd60a4f5b64045953e878fbbbda54c2e40d574215fb515b" gracePeriod=30 Sep 06 14:21:11.534836 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:11.534764 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:11.950206 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:11.950116 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:13.606327 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:13.606281 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:16.718210 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.718180 2577 generic.go:356] "Generic (PLEG): container finished" podID="480b8385-d86f-4f85-a48f-b85de93a36c1" containerID="7d324f274b32b3347dd60a4f5b64045953e878fbbbda54c2e40d574215fb515b" exitCode=0 Sep 06 14:21:16.718613 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.718227 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" event={"ID":"480b8385-d86f-4f85-a48f-b85de93a36c1","Type":"ContainerDied","Data":"7d324f274b32b3347dd60a4f5b64045953e878fbbbda54c2e40d574215fb515b"} Sep 06 14:21:16.718613 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.718290 2577 scope.go:117] "RemoveContainer" containerID="c868992b34d7377e531fa0ea1c5cc6656d2fd0c13b3c6d180e6ba42621f2eccf" Sep 06 14:21:16.776694 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.776635 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:21:16.925511 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.925484 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-envoy\") pod \"480b8385-d86f-4f85-a48f-b85de93a36c1\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " Sep 06 14:21:16.925693 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.925527 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwnr8\" (UniqueName: \"kubernetes.io/projected/480b8385-d86f-4f85-a48f-b85de93a36c1-kube-api-access-zwnr8\") pod \"480b8385-d86f-4f85-a48f-b85de93a36c1\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " Sep 06 14:21:16.925693 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.925550 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-workload-certs\") pod \"480b8385-d86f-4f85-a48f-b85de93a36c1\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " Sep 06 14:21:16.925693 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.925583 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-credential-socket\") pod \"480b8385-d86f-4f85-a48f-b85de93a36c1\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " Sep 06 14:21:16.925693 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.925600 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-token\") pod \"480b8385-d86f-4f85-a48f-b85de93a36c1\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " Sep 06 14:21:16.925693 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.925641 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/480b8385-d86f-4f85-a48f-b85de93a36c1-istiod-ca-cert\") pod \"480b8385-d86f-4f85-a48f-b85de93a36c1\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " Sep 06 14:21:16.925693 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.925690 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-data\") pod \"480b8385-d86f-4f85-a48f-b85de93a36c1\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " Sep 06 14:21:16.926055 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.925756 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-workload-socket\") pod \"480b8385-d86f-4f85-a48f-b85de93a36c1\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " Sep 06 14:21:16.926055 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.925784 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-podinfo\") pod \"480b8385-d86f-4f85-a48f-b85de93a36c1\" (UID: \"480b8385-d86f-4f85-a48f-b85de93a36c1\") " Sep 06 14:21:16.926055 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.925862 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "480b8385-d86f-4f85-a48f-b85de93a36c1" (UID: "480b8385-d86f-4f85-a48f-b85de93a36c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:21:16.926210 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.926084 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-data" (OuterVolumeSpecName: "istio-data") pod "480b8385-d86f-4f85-a48f-b85de93a36c1" (UID: "480b8385-d86f-4f85-a48f-b85de93a36c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:21:16.926210 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.926120 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/480b8385-d86f-4f85-a48f-b85de93a36c1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "480b8385-d86f-4f85-a48f-b85de93a36c1" (UID: "480b8385-d86f-4f85-a48f-b85de93a36c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:21:16.926323 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.926293 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-credential-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:16.926323 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.926301 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "480b8385-d86f-4f85-a48f-b85de93a36c1" (UID: "480b8385-d86f-4f85-a48f-b85de93a36c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:21:16.926323 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.926314 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/480b8385-d86f-4f85-a48f-b85de93a36c1-istiod-ca-cert\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:16.926455 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.926328 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-data\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:16.926455 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.926383 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "480b8385-d86f-4f85-a48f-b85de93a36c1" (UID: "480b8385-d86f-4f85-a48f-b85de93a36c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:21:16.927754 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.927725 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480b8385-d86f-4f85-a48f-b85de93a36c1-kube-api-access-zwnr8" (OuterVolumeSpecName: "kube-api-access-zwnr8") pod "480b8385-d86f-4f85-a48f-b85de93a36c1" (UID: "480b8385-d86f-4f85-a48f-b85de93a36c1"). InnerVolumeSpecName "kube-api-access-zwnr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:21:16.927899 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.927795 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "480b8385-d86f-4f85-a48f-b85de93a36c1" (UID: "480b8385-d86f-4f85-a48f-b85de93a36c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:21:16.928188 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.928164 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-token" (OuterVolumeSpecName: "istio-token") pod "480b8385-d86f-4f85-a48f-b85de93a36c1" (UID: "480b8385-d86f-4f85-a48f-b85de93a36c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:21:16.928265 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:16.928167 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "480b8385-d86f-4f85-a48f-b85de93a36c1" (UID: "480b8385-d86f-4f85-a48f-b85de93a36c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 14:21:17.027732 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:17.027668 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-workload-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:17.027732 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:17.027695 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-podinfo\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:17.027732 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:17.027704 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-envoy\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:17.027732 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:17.027713 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwnr8\" (UniqueName: \"kubernetes.io/projected/480b8385-d86f-4f85-a48f-b85de93a36c1-kube-api-access-zwnr8\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:17.027732 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:17.027721 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/480b8385-d86f-4f85-a48f-b85de93a36c1-workload-certs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:17.027732 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:17.027729 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/480b8385-d86f-4f85-a48f-b85de93a36c1-istio-token\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:17.724674 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:17.724648 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" Sep 06 14:21:17.725127 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:17.724679 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn" event={"ID":"480b8385-d86f-4f85-a48f-b85de93a36c1","Type":"ContainerDied","Data":"aefecee11d6d5011249ea220acb3ee9db8c520deeaccfb46c2dbf59fafb3c8f7"} Sep 06 14:21:17.725127 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:17.724730 2577 scope.go:117] "RemoveContainer" containerID="7d324f274b32b3347dd60a4f5b64045953e878fbbbda54c2e40d574215fb515b" Sep 06 14:21:17.741475 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:17.741453 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn"] Sep 06 14:21:17.744886 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:17.744865 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-61d831-openshift-default-7dbd7954b6-zv2cn"] Sep 06 14:21:19.068072 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:19.068035 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="480b8385-d86f-4f85-a48f-b85de93a36c1" path="/var/lib/kubelet/pods/480b8385-d86f-4f85-a48f-b85de93a36c1/volumes" Sep 06 14:21:22.385204 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:22.385171 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:22.485155 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:22.485131 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:27.314726 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.314693 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:27.353275 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.353228 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf"] Sep 06 14:21:27.353567 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.353539 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" podUID="07caa2e4-73bd-4a30-87b9-9cc23c08df84" containerName="payload-processing" containerID="cri-o://c9cd87dc493e2861513cce673c4d84a01f48910cfe4d14c31e75621e0824e14e" gracePeriod=30 Sep 06 14:21:27.415277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.415239 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:27.480434 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.480407 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:27.530878 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.530835 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:27.620786 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.620761 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" Sep 06 14:21:27.627753 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.627731 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07caa2e4-73bd-4a30-87b9-9cc23c08df84-plugins-config-volume\") pod \"07caa2e4-73bd-4a30-87b9-9cc23c08df84\" (UID: \"07caa2e4-73bd-4a30-87b9-9cc23c08df84\") " Sep 06 14:21:27.627875 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.627778 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgz9n\" (UniqueName: \"kubernetes.io/projected/07caa2e4-73bd-4a30-87b9-9cc23c08df84-kube-api-access-hgz9n\") pod \"07caa2e4-73bd-4a30-87b9-9cc23c08df84\" (UID: \"07caa2e4-73bd-4a30-87b9-9cc23c08df84\") " Sep 06 14:21:27.628112 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.628088 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07caa2e4-73bd-4a30-87b9-9cc23c08df84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07caa2e4-73bd-4a30-87b9-9cc23c08df84" (UID: "07caa2e4-73bd-4a30-87b9-9cc23c08df84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:21:27.629752 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.629730 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07caa2e4-73bd-4a30-87b9-9cc23c08df84-kube-api-access-hgz9n" (OuterVolumeSpecName: "kube-api-access-hgz9n") pod "07caa2e4-73bd-4a30-87b9-9cc23c08df84" (UID: "07caa2e4-73bd-4a30-87b9-9cc23c08df84"). InnerVolumeSpecName "kube-api-access-hgz9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:21:27.728837 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.728805 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07caa2e4-73bd-4a30-87b9-9cc23c08df84-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:27.728837 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.728831 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgz9n\" (UniqueName: \"kubernetes.io/projected/07caa2e4-73bd-4a30-87b9-9cc23c08df84-kube-api-access-hgz9n\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:27.779616 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.779585 2577 generic.go:356] "Generic (PLEG): container finished" podID="07caa2e4-73bd-4a30-87b9-9cc23c08df84" containerID="c9cd87dc493e2861513cce673c4d84a01f48910cfe4d14c31e75621e0824e14e" exitCode=0 Sep 06 14:21:27.779754 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.779635 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" Sep 06 14:21:27.779754 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.779664 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" event={"ID":"07caa2e4-73bd-4a30-87b9-9cc23c08df84","Type":"ContainerDied","Data":"c9cd87dc493e2861513cce673c4d84a01f48910cfe4d14c31e75621e0824e14e"} Sep 06 14:21:27.779754 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.779712 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf" event={"ID":"07caa2e4-73bd-4a30-87b9-9cc23c08df84","Type":"ContainerDied","Data":"fb7d6c62c53b660e3a3c90cf22d8b61d8b0c59a064dc8bd748d5500526fae787"} Sep 06 14:21:27.779754 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.779733 2577 scope.go:117] "RemoveContainer" containerID="c9cd87dc493e2861513cce673c4d84a01f48910cfe4d14c31e75621e0824e14e" Sep 06 14:21:27.789049 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.789024 2577 scope.go:117] "RemoveContainer" containerID="c9cd87dc493e2861513cce673c4d84a01f48910cfe4d14c31e75621e0824e14e" Sep 06 14:21:27.789328 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:21:27.789307 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9cd87dc493e2861513cce673c4d84a01f48910cfe4d14c31e75621e0824e14e\": container with ID starting with c9cd87dc493e2861513cce673c4d84a01f48910cfe4d14c31e75621e0824e14e not found: ID does not exist" containerID="c9cd87dc493e2861513cce673c4d84a01f48910cfe4d14c31e75621e0824e14e" Sep 06 14:21:27.789380 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.789336 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9cd87dc493e2861513cce673c4d84a01f48910cfe4d14c31e75621e0824e14e"} err="failed to get container status \"c9cd87dc493e2861513cce673c4d84a01f48910cfe4d14c31e75621e0824e14e\": rpc error: code = NotFound desc = could not find container \"c9cd87dc493e2861513cce673c4d84a01f48910cfe4d14c31e75621e0824e14e\": container with ID starting with c9cd87dc493e2861513cce673c4d84a01f48910cfe4d14c31e75621e0824e14e not found: ID does not exist" Sep 06 14:21:27.800678 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.800650 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf"] Sep 06 14:21:27.805806 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.805784 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9dd616-f9d69f89-k7rrf"] Sep 06 14:21:27.841416 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:27.841393 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:28.594226 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:28.594192 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:21:29.068524 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:29.068491 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07caa2e4-73bd-4a30-87b9-9cc23c08df84" path="/var/lib/kubelet/pods/07caa2e4-73bd-4a30-87b9-9cc23c08df84/volumes" Sep 06 14:21:33.405828 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:33.405795 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:33.461624 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:33.461594 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:33.646649 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:33.646617 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:34.296822 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:34.296788 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:37.899243 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:37.899209 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:41.296841 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:41.296760 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:21:44.101110 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.101069 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:44.595217 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.595184 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d"] Sep 06 14:21:44.595426 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.595396 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" podUID="32edbadd-fcf9-4db2-946b-6e95600645eb" containerName="maas-api" containerID="cri-o://ef6aa86a4267e061e6410dfb0315b43c508f45fbfa6703bbe7ddaef657f83ca5" gracePeriod=30 Sep 06 14:21:44.611609 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.611581 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:44.637072 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.637010 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr"] Sep 06 14:21:44.637600 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.637291 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" podUID="ffb4890a-ffaf-4cc2-a224-c1a54bd98461" containerName="payload-processing" containerID="cri-o://50fa21284df2cd8f3d76e4601305972d47366c174118ba477d422c5d26411449" gracePeriod=30 Sep 06 14:21:44.644873 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.644847 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7"] Sep 06 14:21:44.645252 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.645222 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" podUID="2f8e593b-f6f4-475e-9327-00454b3af81a" containerName="payload-pre-processing" containerID="cri-o://4802057e59e6e68e20aa5521c767e2fad6cf58f678b91581bd4af5f331910b44" gracePeriod=30 Sep 06 14:21:44.679455 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.678411 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:44.731982 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.731950 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:44.793270 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.793221 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:44.887726 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.887691 2577 generic.go:356] "Generic (PLEG): container finished" podID="32edbadd-fcf9-4db2-946b-6e95600645eb" containerID="ef6aa86a4267e061e6410dfb0315b43c508f45fbfa6703bbe7ddaef657f83ca5" exitCode=0 Sep 06 14:21:44.887901 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.887760 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" event={"ID":"32edbadd-fcf9-4db2-946b-6e95600645eb","Type":"ContainerDied","Data":"ef6aa86a4267e061e6410dfb0315b43c508f45fbfa6703bbe7ddaef657f83ca5"} Sep 06 14:21:44.890182 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.890152 2577 generic.go:356] "Generic (PLEG): container finished" podID="ffb4890a-ffaf-4cc2-a224-c1a54bd98461" containerID="50fa21284df2cd8f3d76e4601305972d47366c174118ba477d422c5d26411449" exitCode=0 Sep 06 14:21:44.890308 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.890256 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" event={"ID":"ffb4890a-ffaf-4cc2-a224-c1a54bd98461","Type":"ContainerDied","Data":"50fa21284df2cd8f3d76e4601305972d47366c174118ba477d422c5d26411449"} Sep 06 14:21:44.894375 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.894350 2577 generic.go:356] "Generic (PLEG): container finished" podID="2f8e593b-f6f4-475e-9327-00454b3af81a" containerID="4802057e59e6e68e20aa5521c767e2fad6cf58f678b91581bd4af5f331910b44" exitCode=0 Sep 06 14:21:44.894503 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.894399 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" event={"ID":"2f8e593b-f6f4-475e-9327-00454b3af81a","Type":"ContainerDied","Data":"4802057e59e6e68e20aa5521c767e2fad6cf58f678b91581bd4af5f331910b44"} Sep 06 14:21:44.950125 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.950097 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" Sep 06 14:21:44.953238 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.953218 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" Sep 06 14:21:44.956798 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:44.956778 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:21:45.086606 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.086577 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32edbadd-fcf9-4db2-946b-6e95600645eb-openshift-service-ca\") pod \"32edbadd-fcf9-4db2-946b-6e95600645eb\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " Sep 06 14:21:45.086765 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.086622 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f8e593b-f6f4-475e-9327-00454b3af81a-plugins-config-volume\") pod \"2f8e593b-f6f4-475e-9327-00454b3af81a\" (UID: \"2f8e593b-f6f4-475e-9327-00454b3af81a\") " Sep 06 14:21:45.086765 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.086716 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32edbadd-fcf9-4db2-946b-6e95600645eb-maas-api-tls\") pod \"32edbadd-fcf9-4db2-946b-6e95600645eb\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " Sep 06 14:21:45.086765 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.086756 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb4890a-ffaf-4cc2-a224-c1a54bd98461-plugins-config-volume\") pod \"ffb4890a-ffaf-4cc2-a224-c1a54bd98461\" (UID: \"ffb4890a-ffaf-4cc2-a224-c1a54bd98461\") " Sep 06 14:21:45.086966 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.086790 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32edbadd-fcf9-4db2-946b-6e95600645eb-metrics-certs\") pod \"32edbadd-fcf9-4db2-946b-6e95600645eb\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " Sep 06 14:21:45.086966 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.086829 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpc9j\" (UniqueName: \"kubernetes.io/projected/2f8e593b-f6f4-475e-9327-00454b3af81a-kube-api-access-cpc9j\") pod \"2f8e593b-f6f4-475e-9327-00454b3af81a\" (UID: \"2f8e593b-f6f4-475e-9327-00454b3af81a\") " Sep 06 14:21:45.086966 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.086888 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs6ln\" (UniqueName: \"kubernetes.io/projected/32edbadd-fcf9-4db2-946b-6e95600645eb-kube-api-access-zs6ln\") pod \"32edbadd-fcf9-4db2-946b-6e95600645eb\" (UID: \"32edbadd-fcf9-4db2-946b-6e95600645eb\") " Sep 06 14:21:45.086966 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.086935 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbsts\" (UniqueName: \"kubernetes.io/projected/ffb4890a-ffaf-4cc2-a224-c1a54bd98461-kube-api-access-cbsts\") pod \"ffb4890a-ffaf-4cc2-a224-c1a54bd98461\" (UID: \"ffb4890a-ffaf-4cc2-a224-c1a54bd98461\") " Sep 06 14:21:45.087183 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.087080 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8e593b-f6f4-475e-9327-00454b3af81a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f8e593b-f6f4-475e-9327-00454b3af81a" (UID: "2f8e593b-f6f4-475e-9327-00454b3af81a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:21:45.087251 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.087223 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb4890a-ffaf-4cc2-a224-c1a54bd98461-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ffb4890a-ffaf-4cc2-a224-c1a54bd98461" (UID: "ffb4890a-ffaf-4cc2-a224-c1a54bd98461"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:21:45.087309 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.087287 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f8e593b-f6f4-475e-9327-00454b3af81a-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:45.087450 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.087415 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32edbadd-fcf9-4db2-946b-6e95600645eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32edbadd-fcf9-4db2-946b-6e95600645eb" (UID: "32edbadd-fcf9-4db2-946b-6e95600645eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:21:45.089947 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.089884 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32edbadd-fcf9-4db2-946b-6e95600645eb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "32edbadd-fcf9-4db2-946b-6e95600645eb" (UID: "32edbadd-fcf9-4db2-946b-6e95600645eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:21:45.090065 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.090031 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8e593b-f6f4-475e-9327-00454b3af81a-kube-api-access-cpc9j" (OuterVolumeSpecName: "kube-api-access-cpc9j") pod "2f8e593b-f6f4-475e-9327-00454b3af81a" (UID: "2f8e593b-f6f4-475e-9327-00454b3af81a"). InnerVolumeSpecName "kube-api-access-cpc9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:21:45.090120 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.090084 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32edbadd-fcf9-4db2-946b-6e95600645eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32edbadd-fcf9-4db2-946b-6e95600645eb" (UID: "32edbadd-fcf9-4db2-946b-6e95600645eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:21:45.090120 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.090096 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb4890a-ffaf-4cc2-a224-c1a54bd98461-kube-api-access-cbsts" (OuterVolumeSpecName: "kube-api-access-cbsts") pod "ffb4890a-ffaf-4cc2-a224-c1a54bd98461" (UID: "ffb4890a-ffaf-4cc2-a224-c1a54bd98461"). InnerVolumeSpecName "kube-api-access-cbsts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:21:45.090247 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.090227 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32edbadd-fcf9-4db2-946b-6e95600645eb-kube-api-access-zs6ln" (OuterVolumeSpecName: "kube-api-access-zs6ln") pod "32edbadd-fcf9-4db2-946b-6e95600645eb" (UID: "32edbadd-fcf9-4db2-946b-6e95600645eb"). InnerVolumeSpecName "kube-api-access-zs6ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:21:45.118079 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.118026 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:45.187868 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.187845 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpc9j\" (UniqueName: \"kubernetes.io/projected/2f8e593b-f6f4-475e-9327-00454b3af81a-kube-api-access-cpc9j\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:45.187868 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.187865 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs6ln\" (UniqueName: \"kubernetes.io/projected/32edbadd-fcf9-4db2-946b-6e95600645eb-kube-api-access-zs6ln\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:45.188012 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.187877 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbsts\" (UniqueName: \"kubernetes.io/projected/ffb4890a-ffaf-4cc2-a224-c1a54bd98461-kube-api-access-cbsts\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:45.188012 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.187885 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32edbadd-fcf9-4db2-946b-6e95600645eb-openshift-service-ca\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:45.188012 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.187895 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32edbadd-fcf9-4db2-946b-6e95600645eb-maas-api-tls\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:45.188012 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.187903 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb4890a-ffaf-4cc2-a224-c1a54bd98461-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:45.188012 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.187929 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32edbadd-fcf9-4db2-946b-6e95600645eb-metrics-certs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:45.904601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.904512 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" Sep 06 14:21:45.904601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.904539 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7" event={"ID":"2f8e593b-f6f4-475e-9327-00454b3af81a","Type":"ContainerDied","Data":"9014730d96db2112a2be31ef4f958c3d1f49e0d9c48d124f7c7f79391d93c5d1"} Sep 06 14:21:45.904601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.904598 2577 scope.go:117] "RemoveContainer" containerID="4802057e59e6e68e20aa5521c767e2fad6cf58f678b91581bd4af5f331910b44" Sep 06 14:21:45.906087 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.906066 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" Sep 06 14:21:45.906214 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.906090 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d" event={"ID":"32edbadd-fcf9-4db2-946b-6e95600645eb","Type":"ContainerDied","Data":"1080dab1b9dff7ecedc7f6221c414876c76485822daa62f25ea4962a48704036"} Sep 06 14:21:45.907612 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.907588 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" Sep 06 14:21:45.907742 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.907608 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr" event={"ID":"ffb4890a-ffaf-4cc2-a224-c1a54bd98461","Type":"ContainerDied","Data":"7af374e89988fad5bd14e40cada85cb1b1ceb648db31b6a6505539a9215d6782"} Sep 06 14:21:45.914294 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.914272 2577 scope.go:117] "RemoveContainer" containerID="ef6aa86a4267e061e6410dfb0315b43c508f45fbfa6703bbe7ddaef657f83ca5" Sep 06 14:21:45.924172 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.924155 2577 scope.go:117] "RemoveContainer" containerID="50fa21284df2cd8f3d76e4601305972d47366c174118ba477d422c5d26411449" Sep 06 14:21:45.931342 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.931322 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7"] Sep 06 14:21:45.935352 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.935330 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6328f9-c4db985cd-2d4r7"] Sep 06 14:21:45.945348 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.945328 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr"] Sep 06 14:21:45.951193 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.951171 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6328f9-897999ffc-g56pr"] Sep 06 14:21:45.960509 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.960488 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d"] Sep 06 14:21:45.964532 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:45.964512 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6328f9-7b69977465-9d98d"] Sep 06 14:21:47.067427 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:47.067390 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f8e593b-f6f4-475e-9327-00454b3af81a" path="/var/lib/kubelet/pods/2f8e593b-f6f4-475e-9327-00454b3af81a/volumes" Sep 06 14:21:47.067797 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:47.067787 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32edbadd-fcf9-4db2-946b-6e95600645eb" path="/var/lib/kubelet/pods/32edbadd-fcf9-4db2-946b-6e95600645eb/volumes" Sep 06 14:21:47.068168 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:47.068154 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffb4890a-ffaf-4cc2-a224-c1a54bd98461" path="/var/lib/kubelet/pods/ffb4890a-ffaf-4cc2-a224-c1a54bd98461/volumes" Sep 06 14:21:49.316709 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:49.316681 2577 scope.go:117] "RemoveContainer" containerID="4ba6e23e4dc2a1e99765008beaef42b6121ba0dc8e37938f86264adb4855a8ad" Sep 06 14:21:49.325044 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:49.325027 2577 scope.go:117] "RemoveContainer" containerID="f266057070026c11586c21949613fdcb41baf7299819a8742f2d727b42392a6b" Sep 06 14:21:50.805411 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:50.805382 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:50.906631 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:50.906593 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:51.056169 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.056088 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:51.393924 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.393831 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn"] Sep 06 14:21:51.394269 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394256 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32edbadd-fcf9-4db2-946b-6e95600645eb" containerName="maas-api" Sep 06 14:21:51.394318 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394271 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="32edbadd-fcf9-4db2-946b-6e95600645eb" containerName="maas-api" Sep 06 14:21:51.394318 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394279 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480b8385-d86f-4f85-a48f-b85de93a36c1" containerName="istio-proxy" Sep 06 14:21:51.394318 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394285 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="480b8385-d86f-4f85-a48f-b85de93a36c1" containerName="istio-proxy" Sep 06 14:21:51.394318 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394295 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07caa2e4-73bd-4a30-87b9-9cc23c08df84" containerName="payload-processing" Sep 06 14:21:51.394318 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394300 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="07caa2e4-73bd-4a30-87b9-9cc23c08df84" containerName="payload-processing" Sep 06 14:21:51.394318 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394312 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480b8385-d86f-4f85-a48f-b85de93a36c1" containerName="istio-proxy" Sep 06 14:21:51.394318 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394317 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="480b8385-d86f-4f85-a48f-b85de93a36c1" containerName="istio-proxy" Sep 06 14:21:51.394526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394329 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffb4890a-ffaf-4cc2-a224-c1a54bd98461" containerName="payload-processing" Sep 06 14:21:51.394526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394335 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffb4890a-ffaf-4cc2-a224-c1a54bd98461" containerName="payload-processing" Sep 06 14:21:51.394526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394345 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f8e593b-f6f4-475e-9327-00454b3af81a" containerName="payload-pre-processing" Sep 06 14:21:51.394526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394350 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f8e593b-f6f4-475e-9327-00454b3af81a" containerName="payload-pre-processing" Sep 06 14:21:51.394526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394361 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d803825f-8585-4db4-b705-c22acaf29a3c" containerName="maas-api" Sep 06 14:21:51.394526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394366 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d803825f-8585-4db4-b705-c22acaf29a3c" containerName="maas-api" Sep 06 14:21:51.394526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394419 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f8e593b-f6f4-475e-9327-00454b3af81a" containerName="payload-pre-processing" Sep 06 14:21:51.394526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394426 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffb4890a-ffaf-4cc2-a224-c1a54bd98461" containerName="payload-processing" Sep 06 14:21:51.394526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394433 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="480b8385-d86f-4f85-a48f-b85de93a36c1" containerName="istio-proxy" Sep 06 14:21:51.394526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394439 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="32edbadd-fcf9-4db2-946b-6e95600645eb" containerName="maas-api" Sep 06 14:21:51.394526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394446 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d803825f-8585-4db4-b705-c22acaf29a3c" containerName="maas-api" Sep 06 14:21:51.394526 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.394452 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="07caa2e4-73bd-4a30-87b9-9cc23c08df84" containerName="payload-processing" Sep 06 14:21:51.399398 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.399373 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn" Sep 06 14:21:51.401569 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.401542 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 14:21:51.401691 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.401606 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\"" Sep 06 14:21:51.403176 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.403152 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn"] Sep 06 14:21:51.546923 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.546878 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee98e29-723d-4dde-af9a-c006eb5a9d99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn\" (UID: \"7ee98e29-723d-4dde-af9a-c006eb5a9d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn" Sep 06 14:21:51.603489 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.603438 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:21:51.648331 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.648257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee98e29-723d-4dde-af9a-c006eb5a9d99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn\" (UID: \"7ee98e29-723d-4dde-af9a-c006eb5a9d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn" Sep 06 14:21:51.648820 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.648800 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee98e29-723d-4dde-af9a-c006eb5a9d99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn\" (UID: \"7ee98e29-723d-4dde-af9a-c006eb5a9d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn" Sep 06 14:21:51.711370 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.711340 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn" Sep 06 14:21:51.982972 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:51.982947 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn"] Sep 06 14:21:51.985251 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:21:51.985226 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee98e29_723d_4dde_af9a_c006eb5a9d99.slice/crio-4a09ea5c5558fca91a6b0ae040781f0971331893c8ee81dbb5a25d7ae6d69247 WatchSource:0}: Error finding container 4a09ea5c5558fca91a6b0ae040781f0971331893c8ee81dbb5a25d7ae6d69247: Status 404 returned error can't find the container with id 4a09ea5c5558fca91a6b0ae040781f0971331893c8ee81dbb5a25d7ae6d69247 Sep 06 14:21:52.950713 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:52.950634 2577 generic.go:356] "Generic (PLEG): container finished" podID="7ee98e29-723d-4dde-af9a-c006eb5a9d99" containerID="0d505d6f3d0e90648c3d49a4b2eb216cb4e55b286b089c80905c23e4c828f5ac" exitCode=0 Sep 06 14:21:52.950713 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:52.950669 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn" event={"ID":"7ee98e29-723d-4dde-af9a-c006eb5a9d99","Type":"ContainerDied","Data":"0d505d6f3d0e90648c3d49a4b2eb216cb4e55b286b089c80905c23e4c828f5ac"} Sep 06 14:21:52.950713 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:52.950706 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn" event={"ID":"7ee98e29-723d-4dde-af9a-c006eb5a9d99","Type":"ContainerStarted","Data":"4a09ea5c5558fca91a6b0ae040781f0971331893c8ee81dbb5a25d7ae6d69247"} Sep 06 14:21:54.079955 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:54.079926 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn" Sep 06 14:21:54.172070 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:54.172043 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee98e29-723d-4dde-af9a-c006eb5a9d99-maas-api-service-ca\") pod \"7ee98e29-723d-4dde-af9a-c006eb5a9d99\" (UID: \"7ee98e29-723d-4dde-af9a-c006eb5a9d99\") " Sep 06 14:21:54.172391 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:54.172361 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee98e29-723d-4dde-af9a-c006eb5a9d99-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ee98e29-723d-4dde-af9a-c006eb5a9d99" (UID: "7ee98e29-723d-4dde-af9a-c006eb5a9d99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:21:54.172927 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:54.172883 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee98e29-723d-4dde-af9a-c006eb5a9d99-maas-api-service-ca\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:21:54.961349 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:54.961315 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn" Sep 06 14:21:54.961349 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:54.961342 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn" event={"ID":"7ee98e29-723d-4dde-af9a-c006eb5a9d99","Type":"ContainerDied","Data":"4a09ea5c5558fca91a6b0ae040781f0971331893c8ee81dbb5a25d7ae6d69247"} Sep 06 14:21:54.961580 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:54.961425 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a09ea5c5558fca91a6b0ae040781f0971331893c8ee81dbb5a25d7ae6d69247" Sep 06 14:21:58.186620 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:21:58.186580 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:22:01.838682 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:01.838634 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:01.888855 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:01.888824 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc"] Sep 06 14:22:01.889111 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:01.889087 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" podUID="a8aade61-f68c-4205-bed5-13acf52493e9" containerName="payload-pre-processing" containerID="cri-o://480911ea9647de2e4ab0216a0ee242070ff9e79aaa4315c878e3b4246a3d7b25" gracePeriod=30 Sep 06 14:22:01.911793 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:01.911756 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:01.984896 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:01.984852 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:02.078598 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:02.078551 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn"] Sep 06 14:22:02.079983 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:02.079938 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:02.080882 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:02.080858 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a388e1-crdnn"] Sep 06 14:22:02.135316 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:02.135281 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:02.137878 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:02.137858 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" Sep 06 14:22:02.247817 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:02.247789 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49vbm\" (UniqueName: \"kubernetes.io/projected/a8aade61-f68c-4205-bed5-13acf52493e9-kube-api-access-49vbm\") pod \"a8aade61-f68c-4205-bed5-13acf52493e9\" (UID: \"a8aade61-f68c-4205-bed5-13acf52493e9\") " Sep 06 14:22:02.247817 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:02.247824 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8aade61-f68c-4205-bed5-13acf52493e9-plugins-config-volume\") pod \"a8aade61-f68c-4205-bed5-13acf52493e9\" (UID: \"a8aade61-f68c-4205-bed5-13acf52493e9\") " Sep 06 14:22:02.248273 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:02.248242 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8aade61-f68c-4205-bed5-13acf52493e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8aade61-f68c-4205-bed5-13acf52493e9" (UID: "a8aade61-f68c-4205-bed5-13acf52493e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:22:02.249953 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:02.249934 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8aade61-f68c-4205-bed5-13acf52493e9-kube-api-access-49vbm" (OuterVolumeSpecName: "kube-api-access-49vbm") pod "a8aade61-f68c-4205-bed5-13acf52493e9" (UID: "a8aade61-f68c-4205-bed5-13acf52493e9"). InnerVolumeSpecName "kube-api-access-49vbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:22:02.348746 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:02.348711 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49vbm\" (UniqueName: \"kubernetes.io/projected/a8aade61-f68c-4205-bed5-13acf52493e9-kube-api-access-49vbm\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:22:02.348746 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:02.348741 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8aade61-f68c-4205-bed5-13acf52493e9-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:22:02.419633 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:02.419543 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:03.001512 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:03.001476 2577 generic.go:356] "Generic (PLEG): container finished" podID="a8aade61-f68c-4205-bed5-13acf52493e9" containerID="480911ea9647de2e4ab0216a0ee242070ff9e79aaa4315c878e3b4246a3d7b25" exitCode=0 Sep 06 14:22:03.001952 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:03.001517 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" Sep 06 14:22:03.001952 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:03.001562 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" event={"ID":"a8aade61-f68c-4205-bed5-13acf52493e9","Type":"ContainerDied","Data":"480911ea9647de2e4ab0216a0ee242070ff9e79aaa4315c878e3b4246a3d7b25"} Sep 06 14:22:03.001952 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:03.001611 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc" event={"ID":"a8aade61-f68c-4205-bed5-13acf52493e9","Type":"ContainerDied","Data":"eba99ff50a80243bd1823dcf0db4f4a78d2bb19e57c15b6ae4dddb82fc59bf8c"} Sep 06 14:22:03.001952 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:03.001636 2577 scope.go:117] "RemoveContainer" containerID="480911ea9647de2e4ab0216a0ee242070ff9e79aaa4315c878e3b4246a3d7b25" Sep 06 14:22:03.011521 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:03.011504 2577 scope.go:117] "RemoveContainer" containerID="480911ea9647de2e4ab0216a0ee242070ff9e79aaa4315c878e3b4246a3d7b25" Sep 06 14:22:03.011774 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:22:03.011756 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480911ea9647de2e4ab0216a0ee242070ff9e79aaa4315c878e3b4246a3d7b25\": container with ID starting with 480911ea9647de2e4ab0216a0ee242070ff9e79aaa4315c878e3b4246a3d7b25 not found: ID does not exist" containerID="480911ea9647de2e4ab0216a0ee242070ff9e79aaa4315c878e3b4246a3d7b25" Sep 06 14:22:03.011820 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:03.011783 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480911ea9647de2e4ab0216a0ee242070ff9e79aaa4315c878e3b4246a3d7b25"} err="failed to get container status \"480911ea9647de2e4ab0216a0ee242070ff9e79aaa4315c878e3b4246a3d7b25\": rpc error: code = NotFound desc = could not find container \"480911ea9647de2e4ab0216a0ee242070ff9e79aaa4315c878e3b4246a3d7b25\": container with ID starting with 480911ea9647de2e4ab0216a0ee242070ff9e79aaa4315c878e3b4246a3d7b25 not found: ID does not exist" Sep 06 14:22:03.023925 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:03.023886 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc"] Sep 06 14:22:03.026306 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:03.026289 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a388e1-c56f54bb-qxrqc"] Sep 06 14:22:03.067215 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:03.067185 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee98e29-723d-4dde-af9a-c006eb5a9d99" path="/var/lib/kubelet/pods/7ee98e29-723d-4dde-af9a-c006eb5a9d99/volumes" Sep 06 14:22:03.067614 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:03.067596 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8aade61-f68c-4205-bed5-13acf52493e9" path="/var/lib/kubelet/pods/a8aade61-f68c-4205-bed5-13acf52493e9/volumes" Sep 06 14:22:08.245430 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:08.245381 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:08.322193 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:08.322128 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:08.494621 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:08.494585 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:08.596120 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:08.596085 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:12.288390 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:12.288353 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:22:17.915675 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:17.915637 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:18.189994 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.189903 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:18.595233 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.595199 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:18.865382 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.865253 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8"] Sep 06 14:22:18.865744 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.865728 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8aade61-f68c-4205-bed5-13acf52493e9" containerName="payload-pre-processing" Sep 06 14:22:18.865796 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.865746 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8aade61-f68c-4205-bed5-13acf52493e9" containerName="payload-pre-processing" Sep 06 14:22:18.865796 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.865777 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee98e29-723d-4dde-af9a-c006eb5a9d99" containerName="revoke-keys" Sep 06 14:22:18.865796 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.865785 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee98e29-723d-4dde-af9a-c006eb5a9d99" containerName="revoke-keys" Sep 06 14:22:18.865892 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.865865 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee98e29-723d-4dde-af9a-c006eb5a9d99" containerName="revoke-keys" Sep 06 14:22:18.865892 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.865874 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8aade61-f68c-4205-bed5-13acf52493e9" containerName="payload-pre-processing" Sep 06 14:22:18.865892 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.865882 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="480b8385-d86f-4f85-a48f-b85de93a36c1" containerName="istio-proxy" Sep 06 14:22:18.869307 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.869291 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.871516 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.871492 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d46936-openshift-default-dockercfg-cr2p4\"" Sep 06 14:22:18.880119 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.880094 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8"] Sep 06 14:22:18.888199 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.888176 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:18.896216 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.896193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-istio-data\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.896343 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.896220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78c162f6-4168-41ef-8c42-dce69c734264-istio-podinfo\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.896343 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.896249 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-workload-certs\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.896343 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.896303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-credential-socket\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.896493 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.896407 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78c162f6-4168-41ef-8c42-dce69c734264-istio-token\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.896493 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.896450 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gsc2\" (UniqueName: \"kubernetes.io/projected/78c162f6-4168-41ef-8c42-dce69c734264-kube-api-access-5gsc2\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.896570 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.896503 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-workload-socket\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.896570 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.896531 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-istio-envoy\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.896694 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.896569 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78c162f6-4168-41ef-8c42-dce69c734264-istiod-ca-cert\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.997876 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.997841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-workload-socket\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.998302 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.997883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-istio-envoy\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.998302 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.997921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78c162f6-4168-41ef-8c42-dce69c734264-istiod-ca-cert\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.998302 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.997962 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-istio-data\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.998302 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.997977 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78c162f6-4168-41ef-8c42-dce69c734264-istio-podinfo\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.998302 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.998005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-workload-certs\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.998302 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.998027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-credential-socket\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.998302 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.998061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78c162f6-4168-41ef-8c42-dce69c734264-istio-token\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.998302 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.998097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gsc2\" (UniqueName: \"kubernetes.io/projected/78c162f6-4168-41ef-8c42-dce69c734264-kube-api-access-5gsc2\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.998757 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.998331 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-workload-socket\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.998757 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.998377 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-istio-data\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.998757 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.998618 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-credential-socket\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.998757 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.998656 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-workload-certs\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:18.999009 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:18.998762 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78c162f6-4168-41ef-8c42-dce69c734264-istiod-ca-cert\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:19.000348 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:19.000323 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-istio-envoy\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:19.000499 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:19.000485 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78c162f6-4168-41ef-8c42-dce69c734264-istio-podinfo\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:19.005220 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:19.005197 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78c162f6-4168-41ef-8c42-dce69c734264-istio-token\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:19.005646 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:19.005626 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gsc2\" (UniqueName: \"kubernetes.io/projected/78c162f6-4168-41ef-8c42-dce69c734264-kube-api-access-5gsc2\") pod \"e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:19.180966 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:19.180862 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:19.455274 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:19.455244 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8"] Sep 06 14:22:19.456379 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:22:19.456356 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c162f6_4168_41ef_8c42_dce69c734264.slice/crio-339ae75baae9c7a0b836f8c2968bd1427f527f76d184c41923bcc1121e431c4a WatchSource:0}: Error finding container 339ae75baae9c7a0b836f8c2968bd1427f527f76d184c41923bcc1121e431c4a: Status 404 returned error can't find the container with id 339ae75baae9c7a0b836f8c2968bd1427f527f76d184c41923bcc1121e431c4a Sep 06 14:22:19.458818 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:19.458779 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:22:19.458930 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:19.458854 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:22:19.458930 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:19.458886 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:22:19.499003 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:19.498970 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:20.092790 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:20.092757 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" event={"ID":"78c162f6-4168-41ef-8c42-dce69c734264","Type":"ContainerStarted","Data":"96c15b713f7b8b1e8508f77375da59b8b139dfbd8bc95449987daf0b2b35d8fa"} Sep 06 14:22:20.093193 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:20.092805 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" event={"ID":"78c162f6-4168-41ef-8c42-dce69c734264","Type":"ContainerStarted","Data":"339ae75baae9c7a0b836f8c2968bd1427f527f76d184c41923bcc1121e431c4a"} Sep 06 14:22:20.112072 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:20.112026 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" podStartSLOduration=2.112011673 podStartE2EDuration="2.112011673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:22:20.108692572 +0000 UTC m=+1471.564181434" watchObservedRunningTime="2026-09-06 14:22:20.112011673 +0000 UTC m=+1471.567500570" Sep 06 14:22:20.181065 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:20.181031 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:20.185751 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:20.185723 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:21.097642 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:21.097612 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:21.098510 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:21.098489 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:21.965180 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:21.965142 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:22.063654 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:22.063626 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:22.887479 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:22.887445 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:22:23.088806 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.088775 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67"] Sep 06 14:22:23.092731 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.092709 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:23.096038 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.096014 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 14:22:23.096192 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.096054 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d46936-serving-cert\"" Sep 06 14:22:23.096192 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.096077 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 06 14:22:23.096192 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.096014 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4lj84\"" Sep 06 14:22:23.096379 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.096241 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 06 14:22:23.102850 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.102614 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67"] Sep 06 14:22:23.120625 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.120601 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:23.136555 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.136531 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:23.136665 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.136638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndrsn\" (UniqueName: \"kubernetes.io/projected/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-kube-api-access-ndrsn\") pod \"maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:23.136712 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.136682 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:23.136772 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.136753 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-metrics-certs\") pod \"maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:23.221597 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.221522 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:23.237481 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.237456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:23.237552 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.237515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndrsn\" (UniqueName: \"kubernetes.io/projected/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-kube-api-access-ndrsn\") pod \"maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:23.237552 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.237542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:23.237665 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.237569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-metrics-certs\") pod \"maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:23.237665 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:22:23.237643 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-serving-cert: secret "maas-api-e2e-ipp-a-d46936-serving-cert" not found Sep 06 14:22:23.237777 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:22:23.237710 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-maas-api-tls podName:d5301ac3-0aeb-43c6-85ec-88b74c67b59c nodeName:}" failed. No retries permitted until 2026-09-06 14:22:23.737693147 +0000 UTC m=+1475.193181988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-maas-api-tls") pod "maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" (UID: "d5301ac3-0aeb-43c6-85ec-88b74c67b59c") : secret "maas-api-e2e-ipp-a-d46936-serving-cert" not found Sep 06 14:22:23.238202 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.238180 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:23.239893 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.239875 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-metrics-certs\") pod \"maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:23.248281 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.248260 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndrsn\" (UniqueName: \"kubernetes.io/projected/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-kube-api-access-ndrsn\") pod \"maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:23.554184 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.554148 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:23.653431 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.653382 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:23.734934 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.734888 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:23.743018 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.742991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:23.745574 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:23.745548 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:24.007704 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:24.007622 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:24.292282 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:24.291968 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67"] Sep 06 14:22:24.294438 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:22:24.294405 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5301ac3_0aeb_43c6_85ec_88b74c67b59c.slice/crio-fce20726c0ae88b1e6eaf7a438ccf84d1145c5b63dcf9cd1a1c6d5ba0bd2152d WatchSource:0}: Error finding container fce20726c0ae88b1e6eaf7a438ccf84d1145c5b63dcf9cd1a1c6d5ba0bd2152d: Status 404 returned error can't find the container with id fce20726c0ae88b1e6eaf7a438ccf84d1145c5b63dcf9cd1a1c6d5ba0bd2152d Sep 06 14:22:24.295934 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:24.295884 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 14:22:25.123512 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:25.123478 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" event={"ID":"d5301ac3-0aeb-43c6-85ec-88b74c67b59c","Type":"ContainerStarted","Data":"53ad8e35debba7e0e698beb22c68cc0c49bf37b3ad45a680fda46ddbf77c1dcf"} Sep 06 14:22:25.123959 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:25.123533 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" event={"ID":"d5301ac3-0aeb-43c6-85ec-88b74c67b59c","Type":"ContainerStarted","Data":"fce20726c0ae88b1e6eaf7a438ccf84d1145c5b63dcf9cd1a1c6d5ba0bd2152d"} Sep 06 14:22:25.123959 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:25.123554 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:25.140034 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:25.139981 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" podStartSLOduration=1.788426944 podStartE2EDuration="2.139964986s" totalImagesPullingTime="351.538042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:22:25.13725177 +0000 UTC m=+1476.592740633" watchObservedRunningTime="2026-09-06 14:22:25.139964986 +0000 UTC m=+1476.595453848" Sep 06 14:22:31.136272 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:31.136240 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:22:31.250434 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:31.250399 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:34.386500 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:34.386462 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:22:40.205838 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:40.205809 2577 generic.go:356] "Generic (PLEG): container finished" podID="78c162f6-4168-41ef-8c42-dce69c734264" containerID="96c15b713f7b8b1e8508f77375da59b8b139dfbd8bc95449987daf0b2b35d8fa" exitCode=0 Sep 06 14:22:40.206255 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:40.205878 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" event={"ID":"78c162f6-4168-41ef-8c42-dce69c734264","Type":"ContainerDied","Data":"96c15b713f7b8b1e8508f77375da59b8b139dfbd8bc95449987daf0b2b35d8fa"} Sep 06 14:22:40.206333 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:40.206319 2577 scope.go:117] "RemoveContainer" containerID="96c15b713f7b8b1e8508f77375da59b8b139dfbd8bc95449987daf0b2b35d8fa" Sep 06 14:22:40.207383 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:40.207353 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:22:40.207483 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:40.207437 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:22:40.207544 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:40.207486 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:22:41.213707 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:41.213666 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" event={"ID":"78c162f6-4168-41ef-8c42-dce69c734264","Type":"ContainerStarted","Data":"68f8dcbf3d62c8ea33b0a05c95667140c70cd3be48f8de31799c729ad5fcdef6"} Sep 06 14:22:42.182047 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:42.182009 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:42.182237 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:42.182213 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 06 14:22:42.182284 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:42.182268 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" podUID="78c162f6-4168-41ef-8c42-dce69c734264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 06 14:22:43.181623 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:43.181582 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 06 14:22:43.182127 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:43.181658 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" podUID="78c162f6-4168-41ef-8c42-dce69c734264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 06 14:22:44.083945 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:44.083885 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:44.181542 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:44.181506 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 06 14:22:44.181707 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:44.181561 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" podUID="78c162f6-4168-41ef-8c42-dce69c734264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 06 14:22:45.181419 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:45.181379 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 06 14:22:45.181589 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:45.181434 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" podUID="78c162f6-4168-41ef-8c42-dce69c734264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 06 14:22:46.360369 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:46.360333 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:46.360811 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:46.360791 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:46.361667 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:46.361650 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:22:46.909779 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:46.909741 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:47.553696 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:47.553666 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:47.851948 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:47.851820 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:48.583558 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:48.583525 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:50.405709 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:50.405671 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:50.505502 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:50.505470 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:51.545459 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:51.545430 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:51.645860 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:51.645830 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:51.768398 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:51.768369 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:52.007522 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:52.007431 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:52.108344 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:52.108312 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:22:52.114666 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:52.114641 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm"] Sep 06 14:22:52.117927 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:52.117899 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" Sep 06 14:22:52.120291 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:52.120275 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c1a83e\"" Sep 06 14:22:52.120374 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:52.120276 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c1a83e-dockercfg-5twrt\"" Sep 06 14:22:52.128693 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:52.128675 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm"] Sep 06 14:22:52.145327 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:52.144993 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041a1de8-413c-43e6-8e8a-52ceb5f3ee4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm\" (UID: \"041a1de8-413c-43e6-8e8a-52ceb5f3ee4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" Sep 06 14:22:52.145984 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:52.145474 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-274p8\" (UniqueName: \"kubernetes.io/projected/041a1de8-413c-43e6-8e8a-52ceb5f3ee4e-kube-api-access-274p8\") pod \"payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm\" (UID: \"041a1de8-413c-43e6-8e8a-52ceb5f3ee4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" Sep 06 14:22:52.248377 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:52.248341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-274p8\" (UniqueName: \"kubernetes.io/projected/041a1de8-413c-43e6-8e8a-52ceb5f3ee4e-kube-api-access-274p8\") pod \"payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm\" (UID: \"041a1de8-413c-43e6-8e8a-52ceb5f3ee4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" Sep 06 14:22:52.248565 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:52.248413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041a1de8-413c-43e6-8e8a-52ceb5f3ee4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm\" (UID: \"041a1de8-413c-43e6-8e8a-52ceb5f3ee4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" Sep 06 14:22:52.249124 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:52.249105 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041a1de8-413c-43e6-8e8a-52ceb5f3ee4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm\" (UID: \"041a1de8-413c-43e6-8e8a-52ceb5f3ee4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" Sep 06 14:22:52.255609 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:52.255583 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-274p8\" (UniqueName: \"kubernetes.io/projected/041a1de8-413c-43e6-8e8a-52ceb5f3ee4e-kube-api-access-274p8\") pod \"payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm\" (UID: \"041a1de8-413c-43e6-8e8a-52ceb5f3ee4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" Sep 06 14:22:52.429317 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:52.429286 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" Sep 06 14:22:52.724621 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:52.716442 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm"] Sep 06 14:22:53.299076 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:53.299041 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" event={"ID":"041a1de8-413c-43e6-8e8a-52ceb5f3ee4e","Type":"ContainerStarted","Data":"9f7e6019e0649c565fcae558c324a2b3ba89c8895ede507871ad2c2ac0cc927c"} Sep 06 14:22:53.299255 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:53.299095 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" event={"ID":"041a1de8-413c-43e6-8e8a-52ceb5f3ee4e","Type":"ContainerStarted","Data":"0fe113e94845cc65bcdc8c80f60c04aa677cb5783ed345e314220b8ce8152c42"} Sep 06 14:22:53.299255 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:53.299132 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" Sep 06 14:22:53.315067 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:53.315016 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" podStartSLOduration=1.315001826 podStartE2EDuration="1.315001826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:22:53.313100431 +0000 UTC m=+1504.768589293" watchObservedRunningTime="2026-09-06 14:22:53.315001826 +0000 UTC m=+1504.770490688" Sep 06 14:22:58.293875 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:22:58.293840 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:23:04.306767 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:23:04.306738 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" Sep 06 14:23:15.359706 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:23:15.359626 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:23:20.255652 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:23:20.255622 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-shz4m_434bf1f2-d6df-4fc0-a8ae-49216214dd8c/payload-processing/0.log" Sep 06 14:23:32.801071 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:23:32.801037 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:23:33.951040 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:23:33.951002 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:23:38.295017 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:23:38.294977 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:23:57.290951 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:23:57.290901 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:24:01.124631 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.124592 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-shz4m_434bf1f2-d6df-4fc0-a8ae-49216214dd8c/payload-processing/0.log" Sep 06 14:24:01.499989 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.499891 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:01.765719 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.765633 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb"] Sep 06 14:24:01.769033 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.768996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.771522 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.771498 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-8e268e-openshift-default-dockercfg-jhmwt\"" Sep 06 14:24:01.783188 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.783164 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb"] Sep 06 14:24:01.793318 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.793295 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:01.894037 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.894001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-data\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.894037 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.894040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z42qw\" (UniqueName: \"kubernetes.io/projected/1c321624-5acb-4712-ba3b-20d5450b1fe6-kube-api-access-z42qw\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.894278 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.894074 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-workload-socket\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.894278 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.894110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-workload-certs\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.894278 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.894147 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-envoy\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.894278 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.894262 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-token\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.894455 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.894345 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-credential-socket\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.894455 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.894363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-podinfo\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.894455 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.894387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c321624-5acb-4712-ba3b-20d5450b1fe6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.995652 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.995622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-token\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.995789 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.995686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-credential-socket\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.995789 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.995719 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-podinfo\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.995789 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.995754 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c321624-5acb-4712-ba3b-20d5450b1fe6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.995936 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.995819 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-data\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.995936 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.995846 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z42qw\" (UniqueName: \"kubernetes.io/projected/1c321624-5acb-4712-ba3b-20d5450b1fe6-kube-api-access-z42qw\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.995936 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.995876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-workload-socket\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.996092 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.995935 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-workload-certs\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.996092 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.995981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-envoy\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.996204 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.996110 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-credential-socket\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.996204 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.996164 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-workload-socket\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.996314 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.996245 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-data\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.996374 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.996316 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-workload-certs\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.996592 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.996575 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c321624-5acb-4712-ba3b-20d5450b1fe6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.998049 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.998025 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-envoy\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:01.998178 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:01.998162 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-podinfo\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:02.002571 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:02.002535 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-token\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:02.002799 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:02.002781 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z42qw\" (UniqueName: \"kubernetes.io/projected/1c321624-5acb-4712-ba3b-20d5450b1fe6-kube-api-access-z42qw\") pod \"e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:02.083400 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:02.083376 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:02.096395 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:02.096372 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:02.355147 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:02.355123 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb"] Sep 06 14:24:02.356978 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:24:02.356947 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c321624_5acb_4712_ba3b_20d5450b1fe6.slice/crio-bab500b1034d500a821e2367baf0e3fdf4d9f243605ab1e138a938e02553ec1f WatchSource:0}: Error finding container bab500b1034d500a821e2367baf0e3fdf4d9f243605ab1e138a938e02553ec1f: Status 404 returned error can't find the container with id bab500b1034d500a821e2367baf0e3fdf4d9f243605ab1e138a938e02553ec1f Sep 06 14:24:02.359446 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:02.359412 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:24:02.359552 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:02.359490 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:24:02.359552 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:02.359534 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:24:02.666098 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:02.666010 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" event={"ID":"1c321624-5acb-4712-ba3b-20d5450b1fe6","Type":"ContainerStarted","Data":"aa1b3c31580d931c6823884ec5200e8ab6dc3cca739f2acdb6db53d108cfb9d7"} Sep 06 14:24:02.666098 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:02.666072 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" event={"ID":"1c321624-5acb-4712-ba3b-20d5450b1fe6","Type":"ContainerStarted","Data":"bab500b1034d500a821e2367baf0e3fdf4d9f243605ab1e138a938e02553ec1f"} Sep 06 14:24:02.684239 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:02.684188 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" podStartSLOduration=1.684174211 podStartE2EDuration="1.684174211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:24:02.682402066 +0000 UTC m=+1574.137890922" watchObservedRunningTime="2026-09-06 14:24:02.684174211 +0000 UTC m=+1574.139663073" Sep 06 14:24:03.084008 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:03.083975 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:03.089359 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:03.089335 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:03.672240 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:03.672207 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:03.673350 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:03.673331 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:04.533636 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:04.533597 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:04.631809 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:04.631774 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:05.836245 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:05.836197 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:05.892022 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:05.891992 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:06.264359 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:06.264282 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:06.369824 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:06.369794 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:06.469371 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:06.469339 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg"] Sep 06 14:24:06.472062 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:06.472044 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" Sep 06 14:24:06.474208 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:06.474188 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8e268e\"" Sep 06 14:24:06.474313 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:06.474229 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-8e268e-dockercfg-bc5z8\"" Sep 06 14:24:06.486578 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:06.486548 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg"] Sep 06 14:24:06.642727 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:06.642691 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d62rh\" (UniqueName: \"kubernetes.io/projected/5c3bd526-2927-4d9a-83ad-b7b3c414e4ff-kube-api-access-d62rh\") pod \"payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg\" (UID: \"5c3bd526-2927-4d9a-83ad-b7b3c414e4ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" Sep 06 14:24:06.642901 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:06.642771 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c3bd526-2927-4d9a-83ad-b7b3c414e4ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg\" (UID: \"5c3bd526-2927-4d9a-83ad-b7b3c414e4ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" Sep 06 14:24:06.743759 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:06.743728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d62rh\" (UniqueName: \"kubernetes.io/projected/5c3bd526-2927-4d9a-83ad-b7b3c414e4ff-kube-api-access-d62rh\") pod \"payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg\" (UID: \"5c3bd526-2927-4d9a-83ad-b7b3c414e4ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" Sep 06 14:24:06.743897 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:06.743787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c3bd526-2927-4d9a-83ad-b7b3c414e4ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg\" (UID: \"5c3bd526-2927-4d9a-83ad-b7b3c414e4ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" Sep 06 14:24:06.744416 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:06.744397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c3bd526-2927-4d9a-83ad-b7b3c414e4ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg\" (UID: \"5c3bd526-2927-4d9a-83ad-b7b3c414e4ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" Sep 06 14:24:06.750857 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:06.750827 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d62rh\" (UniqueName: \"kubernetes.io/projected/5c3bd526-2927-4d9a-83ad-b7b3c414e4ff-kube-api-access-d62rh\") pod \"payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg\" (UID: \"5c3bd526-2927-4d9a-83ad-b7b3c414e4ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" Sep 06 14:24:06.783709 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:06.783687 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" Sep 06 14:24:07.050336 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:07.050309 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg"] Sep 06 14:24:07.052027 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:24:07.051994 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3bd526_2927_4d9a_83ad_b7b3c414e4ff.slice/crio-8de2b3d7a729210d89842a98503052e502df9517860fd5102a374a68e09b533f WatchSource:0}: Error finding container 8de2b3d7a729210d89842a98503052e502df9517860fd5102a374a68e09b533f: Status 404 returned error can't find the container with id 8de2b3d7a729210d89842a98503052e502df9517860fd5102a374a68e09b533f Sep 06 14:24:07.693254 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:07.693217 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" event={"ID":"5c3bd526-2927-4d9a-83ad-b7b3c414e4ff","Type":"ContainerStarted","Data":"4d65bf521c76fb47d7d17cabeac4ac94da210a810ca09a8fe68e415e464c9584"} Sep 06 14:24:07.693433 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:07.693278 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" event={"ID":"5c3bd526-2927-4d9a-83ad-b7b3c414e4ff","Type":"ContainerStarted","Data":"8de2b3d7a729210d89842a98503052e502df9517860fd5102a374a68e09b533f"} Sep 06 14:24:07.693433 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:07.693307 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" Sep 06 14:24:07.708951 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:07.708893 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" podStartSLOduration=1.708878204 podStartE2EDuration="1.708878204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:24:07.70640387 +0000 UTC m=+1579.161892737" watchObservedRunningTime="2026-09-06 14:24:07.708878204 +0000 UTC m=+1579.164367126" Sep 06 14:24:08.334583 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:08.334537 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:08.384184 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:08.384153 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:09.398271 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:09.398238 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:24:18.700477 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:18.700442 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" Sep 06 14:24:19.715032 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:19.714994 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 06 14:24:19.715475 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:19.715063 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" podUID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 06 14:24:21.788058 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:21.788021 2577 generic.go:356] "Generic (PLEG): container finished" podID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerID="aa1b3c31580d931c6823884ec5200e8ab6dc3cca739f2acdb6db53d108cfb9d7" exitCode=0 Sep 06 14:24:21.788449 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:21.788091 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" event={"ID":"1c321624-5acb-4712-ba3b-20d5450b1fe6","Type":"ContainerDied","Data":"aa1b3c31580d931c6823884ec5200e8ab6dc3cca739f2acdb6db53d108cfb9d7"} Sep 06 14:24:21.788511 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:21.788496 2577 scope.go:117] "RemoveContainer" containerID="aa1b3c31580d931c6823884ec5200e8ab6dc3cca739f2acdb6db53d108cfb9d7" Sep 06 14:24:21.789664 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:21.789638 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:24:21.789726 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:21.789706 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:24:21.789763 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:21.789740 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 06 14:24:22.796071 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:22.796032 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" event={"ID":"1c321624-5acb-4712-ba3b-20d5450b1fe6","Type":"ContainerStarted","Data":"91c54c2387c4aa554223586ce4e6ef36f18231111b5d96c76954f8289b4b9312"} Sep 06 14:24:23.083596 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:23.083512 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:23.083767 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:23.083704 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 06 14:24:23.083767 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:23.083752 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" podUID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 06 14:24:24.083971 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:24.083935 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 06 14:24:24.084379 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:24.083998 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" podUID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 06 14:24:25.083773 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:25.083741 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 06 14:24:25.083956 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:25.083809 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" podUID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 06 14:24:26.083990 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:26.083956 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 06 14:24:26.084354 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:26.084008 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" podUID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 06 14:24:27.084498 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:27.084466 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 06 14:24:27.084851 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:27.084515 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" podUID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 06 14:24:28.087941 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:28.087897 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:28.088372 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:28.088153 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:28.088854 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:28.088835 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:29.495199 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:29.495153 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:36.079485 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.079444 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:36.115426 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.115397 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg"] Sep 06 14:24:36.115650 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.115623 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" podUID="5c3bd526-2927-4d9a-83ad-b7b3c414e4ff" containerName="payload-pre-processing" containerID="cri-o://4d65bf521c76fb47d7d17cabeac4ac94da210a810ca09a8fe68e415e464c9584" gracePeriod=30 Sep 06 14:24:36.178484 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.178450 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:36.251166 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.251129 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:36.302738 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.302704 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:36.380210 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.380188 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" Sep 06 14:24:36.559885 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.559850 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d62rh\" (UniqueName: \"kubernetes.io/projected/5c3bd526-2927-4d9a-83ad-b7b3c414e4ff-kube-api-access-d62rh\") pod \"5c3bd526-2927-4d9a-83ad-b7b3c414e4ff\" (UID: \"5c3bd526-2927-4d9a-83ad-b7b3c414e4ff\") " Sep 06 14:24:36.560080 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.559963 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c3bd526-2927-4d9a-83ad-b7b3c414e4ff-plugins-config-volume\") pod \"5c3bd526-2927-4d9a-83ad-b7b3c414e4ff\" (UID: \"5c3bd526-2927-4d9a-83ad-b7b3c414e4ff\") " Sep 06 14:24:36.560339 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.560313 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3bd526-2927-4d9a-83ad-b7b3c414e4ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c3bd526-2927-4d9a-83ad-b7b3c414e4ff" (UID: "5c3bd526-2927-4d9a-83ad-b7b3c414e4ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:24:36.561802 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.561781 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3bd526-2927-4d9a-83ad-b7b3c414e4ff-kube-api-access-d62rh" (OuterVolumeSpecName: "kube-api-access-d62rh") pod "5c3bd526-2927-4d9a-83ad-b7b3c414e4ff" (UID: "5c3bd526-2927-4d9a-83ad-b7b3c414e4ff"). InnerVolumeSpecName "kube-api-access-d62rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:24:36.616549 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.616481 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:36.663568 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.660933 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d62rh\" (UniqueName: \"kubernetes.io/projected/5c3bd526-2927-4d9a-83ad-b7b3c414e4ff-kube-api-access-d62rh\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:24:36.663568 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.660970 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c3bd526-2927-4d9a-83ad-b7b3c414e4ff-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:24:36.877239 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.877145 2577 generic.go:356] "Generic (PLEG): container finished" podID="5c3bd526-2927-4d9a-83ad-b7b3c414e4ff" containerID="4d65bf521c76fb47d7d17cabeac4ac94da210a810ca09a8fe68e415e464c9584" exitCode=0 Sep 06 14:24:36.877410 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.877279 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" event={"ID":"5c3bd526-2927-4d9a-83ad-b7b3c414e4ff","Type":"ContainerDied","Data":"4d65bf521c76fb47d7d17cabeac4ac94da210a810ca09a8fe68e415e464c9584"} Sep 06 14:24:36.877410 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.877289 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" Sep 06 14:24:36.877410 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.877338 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg" event={"ID":"5c3bd526-2927-4d9a-83ad-b7b3c414e4ff","Type":"ContainerDied","Data":"8de2b3d7a729210d89842a98503052e502df9517860fd5102a374a68e09b533f"} Sep 06 14:24:36.877410 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.877368 2577 scope.go:117] "RemoveContainer" containerID="4d65bf521c76fb47d7d17cabeac4ac94da210a810ca09a8fe68e415e464c9584" Sep 06 14:24:36.887528 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.887508 2577 scope.go:117] "RemoveContainer" containerID="4d65bf521c76fb47d7d17cabeac4ac94da210a810ca09a8fe68e415e464c9584" Sep 06 14:24:36.887876 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:24:36.887854 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d65bf521c76fb47d7d17cabeac4ac94da210a810ca09a8fe68e415e464c9584\": container with ID starting with 4d65bf521c76fb47d7d17cabeac4ac94da210a810ca09a8fe68e415e464c9584 not found: ID does not exist" containerID="4d65bf521c76fb47d7d17cabeac4ac94da210a810ca09a8fe68e415e464c9584" Sep 06 14:24:36.887997 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.887885 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d65bf521c76fb47d7d17cabeac4ac94da210a810ca09a8fe68e415e464c9584"} err="failed to get container status \"4d65bf521c76fb47d7d17cabeac4ac94da210a810ca09a8fe68e415e464c9584\": rpc error: code = NotFound desc = could not find container \"4d65bf521c76fb47d7d17cabeac4ac94da210a810ca09a8fe68e415e464c9584\": container with ID starting with 4d65bf521c76fb47d7d17cabeac4ac94da210a810ca09a8fe68e415e464c9584 not found: ID does not exist" Sep 06 14:24:36.902222 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.902181 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg"] Sep 06 14:24:36.904761 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:36.904737 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8e268e-d9d6fccdb-4gqbg"] Sep 06 14:24:37.068480 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:37.068448 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c3bd526-2927-4d9a-83ad-b7b3c414e4ff" path="/var/lib/kubelet/pods/5c3bd526-2927-4d9a-83ad-b7b3c414e4ff/volumes" Sep 06 14:24:41.494922 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:41.491942 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:24:42.421023 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:42.420990 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:42.521154 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:42.521121 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:42.639550 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:42.639513 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb"] Sep 06 14:24:42.639807 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:42.639780 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" podUID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerName="istio-proxy" containerID="cri-o://91c54c2387c4aa554223586ce4e6ef36f18231111b5d96c76954f8289b4b9312" gracePeriod=30 Sep 06 14:24:42.661723 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:42.661697 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:43.090417 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:43.090385 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 06 14:24:43.090580 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:43.090438 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" podUID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 06 14:24:46.415276 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:46.415240 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:24:47.895858 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.895837 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:47.944305 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.944181 2577 generic.go:356] "Generic (PLEG): container finished" podID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerID="91c54c2387c4aa554223586ce4e6ef36f18231111b5d96c76954f8289b4b9312" exitCode=0 Sep 06 14:24:47.944305 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.944224 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" Sep 06 14:24:47.944506 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.944305 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" event={"ID":"1c321624-5acb-4712-ba3b-20d5450b1fe6","Type":"ContainerDied","Data":"91c54c2387c4aa554223586ce4e6ef36f18231111b5d96c76954f8289b4b9312"} Sep 06 14:24:47.944506 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.944357 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb" event={"ID":"1c321624-5acb-4712-ba3b-20d5450b1fe6","Type":"ContainerDied","Data":"bab500b1034d500a821e2367baf0e3fdf4d9f243605ab1e138a938e02553ec1f"} Sep 06 14:24:47.944506 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.944387 2577 scope.go:117] "RemoveContainer" containerID="91c54c2387c4aa554223586ce4e6ef36f18231111b5d96c76954f8289b4b9312" Sep 06 14:24:47.954818 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.954797 2577 scope.go:117] "RemoveContainer" containerID="aa1b3c31580d931c6823884ec5200e8ab6dc3cca739f2acdb6db53d108cfb9d7" Sep 06 14:24:47.964971 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.964952 2577 scope.go:117] "RemoveContainer" containerID="91c54c2387c4aa554223586ce4e6ef36f18231111b5d96c76954f8289b4b9312" Sep 06 14:24:47.965274 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:24:47.965246 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c54c2387c4aa554223586ce4e6ef36f18231111b5d96c76954f8289b4b9312\": container with ID starting with 91c54c2387c4aa554223586ce4e6ef36f18231111b5d96c76954f8289b4b9312 not found: ID does not exist" containerID="91c54c2387c4aa554223586ce4e6ef36f18231111b5d96c76954f8289b4b9312" Sep 06 14:24:47.965349 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.965282 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c54c2387c4aa554223586ce4e6ef36f18231111b5d96c76954f8289b4b9312"} err="failed to get container status \"91c54c2387c4aa554223586ce4e6ef36f18231111b5d96c76954f8289b4b9312\": rpc error: code = NotFound desc = could not find container \"91c54c2387c4aa554223586ce4e6ef36f18231111b5d96c76954f8289b4b9312\": container with ID starting with 91c54c2387c4aa554223586ce4e6ef36f18231111b5d96c76954f8289b4b9312 not found: ID does not exist" Sep 06 14:24:47.965349 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.965301 2577 scope.go:117] "RemoveContainer" containerID="aa1b3c31580d931c6823884ec5200e8ab6dc3cca739f2acdb6db53d108cfb9d7" Sep 06 14:24:47.965608 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:24:47.965590 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1b3c31580d931c6823884ec5200e8ab6dc3cca739f2acdb6db53d108cfb9d7\": container with ID starting with aa1b3c31580d931c6823884ec5200e8ab6dc3cca739f2acdb6db53d108cfb9d7 not found: ID does not exist" containerID="aa1b3c31580d931c6823884ec5200e8ab6dc3cca739f2acdb6db53d108cfb9d7" Sep 06 14:24:47.965659 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.965614 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1b3c31580d931c6823884ec5200e8ab6dc3cca739f2acdb6db53d108cfb9d7"} err="failed to get container status \"aa1b3c31580d931c6823884ec5200e8ab6dc3cca739f2acdb6db53d108cfb9d7\": rpc error: code = NotFound desc = could not find container \"aa1b3c31580d931c6823884ec5200e8ab6dc3cca739f2acdb6db53d108cfb9d7\": container with ID starting with aa1b3c31580d931c6823884ec5200e8ab6dc3cca739f2acdb6db53d108cfb9d7 not found: ID does not exist" Sep 06 14:24:47.966875 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.966854 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-podinfo\") pod \"1c321624-5acb-4712-ba3b-20d5450b1fe6\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " Sep 06 14:24:47.967020 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.966956 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-workload-certs\") pod \"1c321624-5acb-4712-ba3b-20d5450b1fe6\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " Sep 06 14:24:47.967020 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.966986 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-token\") pod \"1c321624-5acb-4712-ba3b-20d5450b1fe6\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " Sep 06 14:24:47.967020 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.967013 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-credential-socket\") pod \"1c321624-5acb-4712-ba3b-20d5450b1fe6\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " Sep 06 14:24:47.967182 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.967065 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z42qw\" (UniqueName: \"kubernetes.io/projected/1c321624-5acb-4712-ba3b-20d5450b1fe6-kube-api-access-z42qw\") pod \"1c321624-5acb-4712-ba3b-20d5450b1fe6\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " Sep 06 14:24:47.967182 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.967099 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-workload-socket\") pod \"1c321624-5acb-4712-ba3b-20d5450b1fe6\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " Sep 06 14:24:47.967182 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.967126 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-data\") pod \"1c321624-5acb-4712-ba3b-20d5450b1fe6\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " Sep 06 14:24:47.967332 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.967201 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-envoy\") pod \"1c321624-5acb-4712-ba3b-20d5450b1fe6\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " Sep 06 14:24:47.967332 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.967265 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c321624-5acb-4712-ba3b-20d5450b1fe6-istiod-ca-cert\") pod \"1c321624-5acb-4712-ba3b-20d5450b1fe6\" (UID: \"1c321624-5acb-4712-ba3b-20d5450b1fe6\") " Sep 06 14:24:47.967650 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.967260 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1c321624-5acb-4712-ba3b-20d5450b1fe6" (UID: "1c321624-5acb-4712-ba3b-20d5450b1fe6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:24:47.967650 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.967518 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1c321624-5acb-4712-ba3b-20d5450b1fe6" (UID: "1c321624-5acb-4712-ba3b-20d5450b1fe6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:24:47.967650 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.967536 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-data" (OuterVolumeSpecName: "istio-data") pod "1c321624-5acb-4712-ba3b-20d5450b1fe6" (UID: "1c321624-5acb-4712-ba3b-20d5450b1fe6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:24:47.967848 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.967673 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1c321624-5acb-4712-ba3b-20d5450b1fe6" (UID: "1c321624-5acb-4712-ba3b-20d5450b1fe6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:24:47.967848 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.967770 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c321624-5acb-4712-ba3b-20d5450b1fe6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1c321624-5acb-4712-ba3b-20d5450b1fe6" (UID: "1c321624-5acb-4712-ba3b-20d5450b1fe6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:24:47.969265 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.969236 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c321624-5acb-4712-ba3b-20d5450b1fe6-kube-api-access-z42qw" (OuterVolumeSpecName: "kube-api-access-z42qw") pod "1c321624-5acb-4712-ba3b-20d5450b1fe6" (UID: "1c321624-5acb-4712-ba3b-20d5450b1fe6"). InnerVolumeSpecName "kube-api-access-z42qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:24:47.969463 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.969435 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-token" (OuterVolumeSpecName: "istio-token") pod "1c321624-5acb-4712-ba3b-20d5450b1fe6" (UID: "1c321624-5acb-4712-ba3b-20d5450b1fe6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:24:47.969560 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.969518 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1c321624-5acb-4712-ba3b-20d5450b1fe6" (UID: "1c321624-5acb-4712-ba3b-20d5450b1fe6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 14:24:47.969560 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:47.969517 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1c321624-5acb-4712-ba3b-20d5450b1fe6" (UID: "1c321624-5acb-4712-ba3b-20d5450b1fe6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:24:48.068261 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:48.068226 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z42qw\" (UniqueName: \"kubernetes.io/projected/1c321624-5acb-4712-ba3b-20d5450b1fe6-kube-api-access-z42qw\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:24:48.068261 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:48.068254 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-workload-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:24:48.068453 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:48.068268 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-data\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:24:48.068453 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:48.068279 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-envoy\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:24:48.068453 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:48.068290 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c321624-5acb-4712-ba3b-20d5450b1fe6-istiod-ca-cert\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:24:48.068453 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:48.068300 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-podinfo\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:24:48.068453 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:48.068309 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-workload-certs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:24:48.068453 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:48.068321 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c321624-5acb-4712-ba3b-20d5450b1fe6-istio-token\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:24:48.068453 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:48.068331 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c321624-5acb-4712-ba3b-20d5450b1fe6-credential-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:24:48.277846 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:48.277813 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb"] Sep 06 14:24:48.283595 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:48.283569 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e268e-openshift-default-75975484b8-vlpsb"] Sep 06 14:24:49.072099 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:24:49.072062 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c321624-5acb-4712-ba3b-20d5450b1fe6" path="/var/lib/kubelet/pods/1c321624-5acb-4712-ba3b-20d5450b1fe6/volumes" Sep 06 14:25:01.572024 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.571989 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67"] Sep 06 14:25:01.572638 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.572580 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" podUID="d5301ac3-0aeb-43c6-85ec-88b74c67b59c" containerName="maas-api" containerID="cri-o://53ad8e35debba7e0e698beb22c68cc0c49bf37b3ad45a680fda46ddbf77c1dcf" gracePeriod=30 Sep 06 14:25:01.600520 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.600473 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:01.661317 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.661290 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:01.724892 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.724862 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:01.786302 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.786271 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:01.825296 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.825241 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:25:01.895170 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.895141 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndrsn\" (UniqueName: \"kubernetes.io/projected/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-kube-api-access-ndrsn\") pod \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " Sep 06 14:25:01.895365 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.895193 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-maas-api-tls\") pod \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " Sep 06 14:25:01.895431 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.895373 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-metrics-certs\") pod \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " Sep 06 14:25:01.895431 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.895425 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-openshift-service-ca\") pod \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\" (UID: \"d5301ac3-0aeb-43c6-85ec-88b74c67b59c\") " Sep 06 14:25:01.895798 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.895761 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5301ac3-0aeb-43c6-85ec-88b74c67b59c" (UID: "d5301ac3-0aeb-43c6-85ec-88b74c67b59c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:25:01.897448 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.897423 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d5301ac3-0aeb-43c6-85ec-88b74c67b59c" (UID: "d5301ac3-0aeb-43c6-85ec-88b74c67b59c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:25:01.897793 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.897769 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5301ac3-0aeb-43c6-85ec-88b74c67b59c" (UID: "d5301ac3-0aeb-43c6-85ec-88b74c67b59c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:25:01.897887 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.897796 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-kube-api-access-ndrsn" (OuterVolumeSpecName: "kube-api-access-ndrsn") pod "d5301ac3-0aeb-43c6-85ec-88b74c67b59c" (UID: "d5301ac3-0aeb-43c6-85ec-88b74c67b59c"). InnerVolumeSpecName "kube-api-access-ndrsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:25:01.996414 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.996384 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-metrics-certs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:25:01.996414 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.996411 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-openshift-service-ca\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:25:01.996605 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.996423 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndrsn\" (UniqueName: \"kubernetes.io/projected/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-kube-api-access-ndrsn\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:25:01.996605 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:01.996432 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5301ac3-0aeb-43c6-85ec-88b74c67b59c-maas-api-tls\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:25:02.021338 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:02.021309 2577 generic.go:356] "Generic (PLEG): container finished" podID="d5301ac3-0aeb-43c6-85ec-88b74c67b59c" containerID="53ad8e35debba7e0e698beb22c68cc0c49bf37b3ad45a680fda46ddbf77c1dcf" exitCode=0 Sep 06 14:25:02.021469 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:02.021355 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" Sep 06 14:25:02.021469 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:02.021364 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" event={"ID":"d5301ac3-0aeb-43c6-85ec-88b74c67b59c","Type":"ContainerDied","Data":"53ad8e35debba7e0e698beb22c68cc0c49bf37b3ad45a680fda46ddbf77c1dcf"} Sep 06 14:25:02.021469 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:02.021405 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67" event={"ID":"d5301ac3-0aeb-43c6-85ec-88b74c67b59c","Type":"ContainerDied","Data":"fce20726c0ae88b1e6eaf7a438ccf84d1145c5b63dcf9cd1a1c6d5ba0bd2152d"} Sep 06 14:25:02.021469 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:02.021427 2577 scope.go:117] "RemoveContainer" containerID="53ad8e35debba7e0e698beb22c68cc0c49bf37b3ad45a680fda46ddbf77c1dcf" Sep 06 14:25:02.030670 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:02.030654 2577 scope.go:117] "RemoveContainer" containerID="53ad8e35debba7e0e698beb22c68cc0c49bf37b3ad45a680fda46ddbf77c1dcf" Sep 06 14:25:02.031029 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:25:02.031007 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ad8e35debba7e0e698beb22c68cc0c49bf37b3ad45a680fda46ddbf77c1dcf\": container with ID starting with 53ad8e35debba7e0e698beb22c68cc0c49bf37b3ad45a680fda46ddbf77c1dcf not found: ID does not exist" containerID="53ad8e35debba7e0e698beb22c68cc0c49bf37b3ad45a680fda46ddbf77c1dcf" Sep 06 14:25:02.031123 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:02.031042 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ad8e35debba7e0e698beb22c68cc0c49bf37b3ad45a680fda46ddbf77c1dcf"} err="failed to get container status \"53ad8e35debba7e0e698beb22c68cc0c49bf37b3ad45a680fda46ddbf77c1dcf\": rpc error: code = NotFound desc = could not find container \"53ad8e35debba7e0e698beb22c68cc0c49bf37b3ad45a680fda46ddbf77c1dcf\": container with ID starting with 53ad8e35debba7e0e698beb22c68cc0c49bf37b3ad45a680fda46ddbf77c1dcf not found: ID does not exist" Sep 06 14:25:02.041599 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:02.041577 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67"] Sep 06 14:25:02.045025 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:02.045003 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d46936-596d485ff9-z6m67"] Sep 06 14:25:02.119802 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:02.119714 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:03.067833 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:03.067799 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5301ac3-0aeb-43c6-85ec-88b74c67b59c" path="/var/lib/kubelet/pods/d5301ac3-0aeb-43c6-85ec-88b74c67b59c/volumes" Sep 06 14:25:03.187152 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:03.187120 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:25:07.198058 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:07.198025 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:07.842448 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:07.842419 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:07.893306 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:07.893276 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:12.996302 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:12.996262 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:25:21.534659 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:21.534620 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:43.956841 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:43.956804 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:44.175466 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:44.175437 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:49.408081 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:49.408047 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:49.928826 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:49.928788 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:49.987414 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:49.987383 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:50.174856 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:50.174824 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8"] Sep 06 14:25:50.175098 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:50.175073 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" podUID="78c162f6-4168-41ef-8c42-dce69c734264" containerName="istio-proxy" containerID="cri-o://68f8dcbf3d62c8ea33b0a05c95667140c70cd3be48f8de31799c729ad5fcdef6" gracePeriod=30 Sep 06 14:25:50.196448 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:50.196370 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:50.626969 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:50.626933 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:25:55.301717 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.301684 2577 generic.go:356] "Generic (PLEG): container finished" podID="78c162f6-4168-41ef-8c42-dce69c734264" containerID="68f8dcbf3d62c8ea33b0a05c95667140c70cd3be48f8de31799c729ad5fcdef6" exitCode=0 Sep 06 14:25:55.302133 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.301714 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" event={"ID":"78c162f6-4168-41ef-8c42-dce69c734264","Type":"ContainerDied","Data":"68f8dcbf3d62c8ea33b0a05c95667140c70cd3be48f8de31799c729ad5fcdef6"} Sep 06 14:25:55.302133 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.301788 2577 scope.go:117] "RemoveContainer" containerID="96c15b713f7b8b1e8508f77375da59b8b139dfbd8bc95449987daf0b2b35d8fa" Sep 06 14:25:55.432243 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.432220 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:25:55.588164 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.588138 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78c162f6-4168-41ef-8c42-dce69c734264-istio-token\") pod \"78c162f6-4168-41ef-8c42-dce69c734264\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " Sep 06 14:25:55.588335 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.588175 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-workload-socket\") pod \"78c162f6-4168-41ef-8c42-dce69c734264\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " Sep 06 14:25:55.588335 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.588219 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-istio-envoy\") pod \"78c162f6-4168-41ef-8c42-dce69c734264\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " Sep 06 14:25:55.588335 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.588263 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-credential-socket\") pod \"78c162f6-4168-41ef-8c42-dce69c734264\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " Sep 06 14:25:55.588335 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.588324 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gsc2\" (UniqueName: \"kubernetes.io/projected/78c162f6-4168-41ef-8c42-dce69c734264-kube-api-access-5gsc2\") pod \"78c162f6-4168-41ef-8c42-dce69c734264\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " Sep 06 14:25:55.588547 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.588351 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-workload-certs\") pod \"78c162f6-4168-41ef-8c42-dce69c734264\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " Sep 06 14:25:55.588547 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.588378 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78c162f6-4168-41ef-8c42-dce69c734264-istio-podinfo\") pod \"78c162f6-4168-41ef-8c42-dce69c734264\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " Sep 06 14:25:55.588547 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.588416 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-istio-data\") pod \"78c162f6-4168-41ef-8c42-dce69c734264\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " Sep 06 14:25:55.588547 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.588461 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78c162f6-4168-41ef-8c42-dce69c734264-istiod-ca-cert\") pod \"78c162f6-4168-41ef-8c42-dce69c734264\" (UID: \"78c162f6-4168-41ef-8c42-dce69c734264\") " Sep 06 14:25:55.588751 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.588531 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "78c162f6-4168-41ef-8c42-dce69c734264" (UID: "78c162f6-4168-41ef-8c42-dce69c734264"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:25:55.588751 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.588608 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "78c162f6-4168-41ef-8c42-dce69c734264" (UID: "78c162f6-4168-41ef-8c42-dce69c734264"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:25:55.588751 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.588629 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "78c162f6-4168-41ef-8c42-dce69c734264" (UID: "78c162f6-4168-41ef-8c42-dce69c734264"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:25:55.588933 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.588816 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-istio-data" (OuterVolumeSpecName: "istio-data") pod "78c162f6-4168-41ef-8c42-dce69c734264" (UID: "78c162f6-4168-41ef-8c42-dce69c734264"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:25:55.589004 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.588898 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c162f6-4168-41ef-8c42-dce69c734264-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "78c162f6-4168-41ef-8c42-dce69c734264" (UID: "78c162f6-4168-41ef-8c42-dce69c734264"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:25:55.589065 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.589008 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-workload-certs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:25:55.589065 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.589027 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-istio-data\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:25:55.589167 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.589042 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-workload-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:25:55.589167 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.589082 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-credential-socket\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:25:55.590487 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.590453 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "78c162f6-4168-41ef-8c42-dce69c734264" (UID: "78c162f6-4168-41ef-8c42-dce69c734264"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:25:55.590608 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.590531 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78c162f6-4168-41ef-8c42-dce69c734264-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "78c162f6-4168-41ef-8c42-dce69c734264" (UID: "78c162f6-4168-41ef-8c42-dce69c734264"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 14:25:55.590608 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.590557 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c162f6-4168-41ef-8c42-dce69c734264-istio-token" (OuterVolumeSpecName: "istio-token") pod "78c162f6-4168-41ef-8c42-dce69c734264" (UID: "78c162f6-4168-41ef-8c42-dce69c734264"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:25:55.590608 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.590557 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c162f6-4168-41ef-8c42-dce69c734264-kube-api-access-5gsc2" (OuterVolumeSpecName: "kube-api-access-5gsc2") pod "78c162f6-4168-41ef-8c42-dce69c734264" (UID: "78c162f6-4168-41ef-8c42-dce69c734264"). InnerVolumeSpecName "kube-api-access-5gsc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:25:55.689640 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.689615 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78c162f6-4168-41ef-8c42-dce69c734264-istio-token\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:25:55.689640 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.689641 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78c162f6-4168-41ef-8c42-dce69c734264-istio-envoy\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:25:55.689830 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.689654 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gsc2\" (UniqueName: \"kubernetes.io/projected/78c162f6-4168-41ef-8c42-dce69c734264-kube-api-access-5gsc2\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:25:55.689830 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.689664 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78c162f6-4168-41ef-8c42-dce69c734264-istio-podinfo\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:25:55.689830 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:55.689672 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78c162f6-4168-41ef-8c42-dce69c734264-istiod-ca-cert\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:25:56.307749 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:56.307723 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" Sep 06 14:25:56.308184 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:56.307763 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8" event={"ID":"78c162f6-4168-41ef-8c42-dce69c734264","Type":"ContainerDied","Data":"339ae75baae9c7a0b836f8c2968bd1427f527f76d184c41923bcc1121e431c4a"} Sep 06 14:25:56.308184 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:56.307820 2577 scope.go:117] "RemoveContainer" containerID="68f8dcbf3d62c8ea33b0a05c95667140c70cd3be48f8de31799c729ad5fcdef6" Sep 06 14:25:56.328160 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:56.328134 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8"] Sep 06 14:25:56.333187 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:56.333163 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d46936-openshift-default-585db4d5d6-225n8"] Sep 06 14:25:57.067886 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:25:57.067852 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78c162f6-4168-41ef-8c42-dce69c734264" path="/var/lib/kubelet/pods/78c162f6-4168-41ef-8c42-dce69c734264/volumes" Sep 06 14:26:00.966533 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:00.966500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:00.997668 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:00.997639 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm"] Sep 06 14:26:00.997875 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:00.997849 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" podUID="041a1de8-413c-43e6-8e8a-52ceb5f3ee4e" containerName="payload-pre-processing" containerID="cri-o://9f7e6019e0649c565fcae558c324a2b3ba89c8895ede507871ad2c2ac0cc927c" gracePeriod=30 Sep 06 14:26:01.066656 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.066625 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:01.133818 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.133774 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:01.193440 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.193406 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:01.267456 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.267430 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" Sep 06 14:26:01.335687 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.335645 2577 generic.go:356] "Generic (PLEG): container finished" podID="041a1de8-413c-43e6-8e8a-52ceb5f3ee4e" containerID="9f7e6019e0649c565fcae558c324a2b3ba89c8895ede507871ad2c2ac0cc927c" exitCode=0 Sep 06 14:26:01.335687 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.335687 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" Sep 06 14:26:01.335962 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.335732 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" event={"ID":"041a1de8-413c-43e6-8e8a-52ceb5f3ee4e","Type":"ContainerDied","Data":"9f7e6019e0649c565fcae558c324a2b3ba89c8895ede507871ad2c2ac0cc927c"} Sep 06 14:26:01.335962 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.335788 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm" event={"ID":"041a1de8-413c-43e6-8e8a-52ceb5f3ee4e","Type":"ContainerDied","Data":"0fe113e94845cc65bcdc8c80f60c04aa677cb5783ed345e314220b8ce8152c42"} Sep 06 14:26:01.335962 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.335820 2577 scope.go:117] "RemoveContainer" containerID="9f7e6019e0649c565fcae558c324a2b3ba89c8895ede507871ad2c2ac0cc927c" Sep 06 14:26:01.340091 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.339933 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041a1de8-413c-43e6-8e8a-52ceb5f3ee4e-plugins-config-volume\") pod \"041a1de8-413c-43e6-8e8a-52ceb5f3ee4e\" (UID: \"041a1de8-413c-43e6-8e8a-52ceb5f3ee4e\") " Sep 06 14:26:01.340091 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.340047 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-274p8\" (UniqueName: \"kubernetes.io/projected/041a1de8-413c-43e6-8e8a-52ceb5f3ee4e-kube-api-access-274p8\") pod \"041a1de8-413c-43e6-8e8a-52ceb5f3ee4e\" (UID: \"041a1de8-413c-43e6-8e8a-52ceb5f3ee4e\") " Sep 06 14:26:01.340416 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.340390 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/041a1de8-413c-43e6-8e8a-52ceb5f3ee4e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "041a1de8-413c-43e6-8e8a-52ceb5f3ee4e" (UID: "041a1de8-413c-43e6-8e8a-52ceb5f3ee4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:26:01.343409 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.343248 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041a1de8-413c-43e6-8e8a-52ceb5f3ee4e-kube-api-access-274p8" (OuterVolumeSpecName: "kube-api-access-274p8") pod "041a1de8-413c-43e6-8e8a-52ceb5f3ee4e" (UID: "041a1de8-413c-43e6-8e8a-52ceb5f3ee4e"). InnerVolumeSpecName "kube-api-access-274p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:26:01.345970 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.345952 2577 scope.go:117] "RemoveContainer" containerID="9f7e6019e0649c565fcae558c324a2b3ba89c8895ede507871ad2c2ac0cc927c" Sep 06 14:26:01.346281 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:26:01.346252 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f7e6019e0649c565fcae558c324a2b3ba89c8895ede507871ad2c2ac0cc927c\": container with ID starting with 9f7e6019e0649c565fcae558c324a2b3ba89c8895ede507871ad2c2ac0cc927c not found: ID does not exist" containerID="9f7e6019e0649c565fcae558c324a2b3ba89c8895ede507871ad2c2ac0cc927c" Sep 06 14:26:01.346376 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.346288 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f7e6019e0649c565fcae558c324a2b3ba89c8895ede507871ad2c2ac0cc927c"} err="failed to get container status \"9f7e6019e0649c565fcae558c324a2b3ba89c8895ede507871ad2c2ac0cc927c\": rpc error: code = NotFound desc = could not find container \"9f7e6019e0649c565fcae558c324a2b3ba89c8895ede507871ad2c2ac0cc927c\": container with ID starting with 9f7e6019e0649c565fcae558c324a2b3ba89c8895ede507871ad2c2ac0cc927c not found: ID does not exist" Sep 06 14:26:01.441743 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.441710 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041a1de8-413c-43e6-8e8a-52ceb5f3ee4e-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:26:01.441743 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.441738 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-274p8\" (UniqueName: \"kubernetes.io/projected/041a1de8-413c-43e6-8e8a-52ceb5f3ee4e-kube-api-access-274p8\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:26:01.503021 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.502947 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:01.656405 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.656374 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm"] Sep 06 14:26:01.659798 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:01.659773 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1a83e-68ff9f897d-qrgxm"] Sep 06 14:26:03.068248 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:03.068214 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="041a1de8-413c-43e6-8e8a-52ceb5f3ee4e" path="/var/lib/kubelet/pods/041a1de8-413c-43e6-8e8a-52ceb5f3ee4e/volumes" Sep 06 14:26:05.593434 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:05.593398 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:26:07.163859 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:07.163826 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:07.287838 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:07.287801 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:07.441486 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:07.441406 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:07.992757 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:07.992721 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:18.272390 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.272295 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:18.300897 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.300863 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh"] Sep 06 14:26:18.301152 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.301125 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" podUID="f7da9281-f964-4eae-a5a7-c2cab274009a" containerName="payload-pre-processing" containerID="cri-o://d9911fa92dcb5e86be35db635dce217c9931cfbc50321e48428660b9dbfde4d5" gracePeriod=30 Sep 06 14:26:18.371385 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.371349 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:18.426977 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.426949 2577 generic.go:356] "Generic (PLEG): container finished" podID="f7da9281-f964-4eae-a5a7-c2cab274009a" containerID="d9911fa92dcb5e86be35db635dce217c9931cfbc50321e48428660b9dbfde4d5" exitCode=0 Sep 06 14:26:18.427127 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.427019 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" event={"ID":"f7da9281-f964-4eae-a5a7-c2cab274009a","Type":"ContainerDied","Data":"d9911fa92dcb5e86be35db635dce217c9931cfbc50321e48428660b9dbfde4d5"} Sep 06 14:26:18.438158 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.438116 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:18.489789 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.489762 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:18.571838 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.571809 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" Sep 06 14:26:18.603059 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.603032 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvqtg\" (UniqueName: \"kubernetes.io/projected/f7da9281-f964-4eae-a5a7-c2cab274009a-kube-api-access-cvqtg\") pod \"f7da9281-f964-4eae-a5a7-c2cab274009a\" (UID: \"f7da9281-f964-4eae-a5a7-c2cab274009a\") " Sep 06 14:26:18.603211 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.603094 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7da9281-f964-4eae-a5a7-c2cab274009a-plugins-config-volume\") pod \"f7da9281-f964-4eae-a5a7-c2cab274009a\" (UID: \"f7da9281-f964-4eae-a5a7-c2cab274009a\") " Sep 06 14:26:18.603483 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.603455 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7da9281-f964-4eae-a5a7-c2cab274009a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7da9281-f964-4eae-a5a7-c2cab274009a" (UID: "f7da9281-f964-4eae-a5a7-c2cab274009a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:26:18.605102 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.605078 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7da9281-f964-4eae-a5a7-c2cab274009a-kube-api-access-cvqtg" (OuterVolumeSpecName: "kube-api-access-cvqtg") pod "f7da9281-f964-4eae-a5a7-c2cab274009a" (UID: "f7da9281-f964-4eae-a5a7-c2cab274009a"). InnerVolumeSpecName "kube-api-access-cvqtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:26:18.704756 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.704724 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7da9281-f964-4eae-a5a7-c2cab274009a-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:26:18.704756 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.704751 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvqtg\" (UniqueName: \"kubernetes.io/projected/f7da9281-f964-4eae-a5a7-c2cab274009a-kube-api-access-cvqtg\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:26:18.807895 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:18.807797 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:19.434008 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:19.433928 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" Sep 06 14:26:19.434008 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:19.433956 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh" event={"ID":"f7da9281-f964-4eae-a5a7-c2cab274009a","Type":"ContainerDied","Data":"d2df82319b56d4e55a18e607d802063d330e2b7bc6fd2c1161cd9982a811513e"} Sep 06 14:26:19.434508 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:19.434016 2577 scope.go:117] "RemoveContainer" containerID="d9911fa92dcb5e86be35db635dce217c9931cfbc50321e48428660b9dbfde4d5" Sep 06 14:26:19.450297 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:19.450272 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh"] Sep 06 14:26:19.454432 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:19.454411 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0aaff3-756d75f5fd-ngdfh"] Sep 06 14:26:21.068248 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:21.068219 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7da9281-f964-4eae-a5a7-c2cab274009a" path="/var/lib/kubelet/pods/f7da9281-f964-4eae-a5a7-c2cab274009a/volumes" Sep 06 14:26:22.091707 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:22.091671 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:26:24.547486 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:24.547446 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:24.599565 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:24.599517 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:24.804750 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:24.804663 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:25.240339 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:25.240240 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:35.574119 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.574080 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:35.602580 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.602543 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq"] Sep 06 14:26:35.602799 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.602753 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" podUID="9ea60091-a515-422f-8cde-42894f870611" containerName="payload-processing" containerID="cri-o://eeb492cf7fea75b6e5407ab22cac4c8183134da2831a8d639b2765cedbf96402" gracePeriod=30 Sep 06 14:26:35.614286 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.614259 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2"] Sep 06 14:26:35.614504 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.614481 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" podUID="0608cdb9-87d3-4a8d-9c52-e5246d8b38e2" containerName="payload-pre-processing" containerID="cri-o://0e86e0b8a3a026b1fbf0f3738b0331771f9aeb85bd7a8c721d307d58f60a220c" gracePeriod=30 Sep 06 14:26:35.640793 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.640605 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:35.699191 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.699160 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:35.755210 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.755138 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:35.892946 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.892901 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" Sep 06 14:26:35.895971 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.895952 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" Sep 06 14:26:35.972864 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.972836 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea60091-a515-422f-8cde-42894f870611-plugins-config-volume\") pod \"9ea60091-a515-422f-8cde-42894f870611\" (UID: \"9ea60091-a515-422f-8cde-42894f870611\") " Sep 06 14:26:35.973042 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.972890 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fnws\" (UniqueName: \"kubernetes.io/projected/0608cdb9-87d3-4a8d-9c52-e5246d8b38e2-kube-api-access-4fnws\") pod \"0608cdb9-87d3-4a8d-9c52-e5246d8b38e2\" (UID: \"0608cdb9-87d3-4a8d-9c52-e5246d8b38e2\") " Sep 06 14:26:35.973116 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.973074 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6spc\" (UniqueName: \"kubernetes.io/projected/9ea60091-a515-422f-8cde-42894f870611-kube-api-access-q6spc\") pod \"9ea60091-a515-422f-8cde-42894f870611\" (UID: \"9ea60091-a515-422f-8cde-42894f870611\") " Sep 06 14:26:35.973175 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.973117 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0608cdb9-87d3-4a8d-9c52-e5246d8b38e2-plugins-config-volume\") pod \"0608cdb9-87d3-4a8d-9c52-e5246d8b38e2\" (UID: \"0608cdb9-87d3-4a8d-9c52-e5246d8b38e2\") " Sep 06 14:26:35.973261 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.973234 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea60091-a515-422f-8cde-42894f870611-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ea60091-a515-422f-8cde-42894f870611" (UID: "9ea60091-a515-422f-8cde-42894f870611"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:26:35.973459 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.973439 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea60091-a515-422f-8cde-42894f870611-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:26:35.973562 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.973474 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0608cdb9-87d3-4a8d-9c52-e5246d8b38e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0608cdb9-87d3-4a8d-9c52-e5246d8b38e2" (UID: "0608cdb9-87d3-4a8d-9c52-e5246d8b38e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:26:35.975002 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.974977 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0608cdb9-87d3-4a8d-9c52-e5246d8b38e2-kube-api-access-4fnws" (OuterVolumeSpecName: "kube-api-access-4fnws") pod "0608cdb9-87d3-4a8d-9c52-e5246d8b38e2" (UID: "0608cdb9-87d3-4a8d-9c52-e5246d8b38e2"). InnerVolumeSpecName "kube-api-access-4fnws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:26:35.975090 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:35.975073 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea60091-a515-422f-8cde-42894f870611-kube-api-access-q6spc" (OuterVolumeSpecName: "kube-api-access-q6spc") pod "9ea60091-a515-422f-8cde-42894f870611" (UID: "9ea60091-a515-422f-8cde-42894f870611"). InnerVolumeSpecName "kube-api-access-q6spc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:26:36.074403 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.074378 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fnws\" (UniqueName: \"kubernetes.io/projected/0608cdb9-87d3-4a8d-9c52-e5246d8b38e2-kube-api-access-4fnws\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:26:36.074403 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.074398 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6spc\" (UniqueName: \"kubernetes.io/projected/9ea60091-a515-422f-8cde-42894f870611-kube-api-access-q6spc\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:26:36.074545 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.074409 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0608cdb9-87d3-4a8d-9c52-e5246d8b38e2-plugins-config-volume\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:26:36.102434 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.102409 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:36.521506 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.521466 2577 generic.go:356] "Generic (PLEG): container finished" podID="0608cdb9-87d3-4a8d-9c52-e5246d8b38e2" containerID="0e86e0b8a3a026b1fbf0f3738b0331771f9aeb85bd7a8c721d307d58f60a220c" exitCode=0 Sep 06 14:26:36.521506 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.521507 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" Sep 06 14:26:36.521727 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.521548 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" event={"ID":"0608cdb9-87d3-4a8d-9c52-e5246d8b38e2","Type":"ContainerDied","Data":"0e86e0b8a3a026b1fbf0f3738b0331771f9aeb85bd7a8c721d307d58f60a220c"} Sep 06 14:26:36.521727 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.521599 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2" event={"ID":"0608cdb9-87d3-4a8d-9c52-e5246d8b38e2","Type":"ContainerDied","Data":"ae41734a1f35a91da699f54c1cca61e5d38527a4d33eefd210abaea1d84ddb18"} Sep 06 14:26:36.521727 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.521621 2577 scope.go:117] "RemoveContainer" containerID="0e86e0b8a3a026b1fbf0f3738b0331771f9aeb85bd7a8c721d307d58f60a220c" Sep 06 14:26:36.523008 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.522987 2577 generic.go:356] "Generic (PLEG): container finished" podID="9ea60091-a515-422f-8cde-42894f870611" containerID="eeb492cf7fea75b6e5407ab22cac4c8183134da2831a8d639b2765cedbf96402" exitCode=0 Sep 06 14:26:36.523134 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.523015 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" event={"ID":"9ea60091-a515-422f-8cde-42894f870611","Type":"ContainerDied","Data":"eeb492cf7fea75b6e5407ab22cac4c8183134da2831a8d639b2765cedbf96402"} Sep 06 14:26:36.523134 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.523036 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" Sep 06 14:26:36.523134 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.523056 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq" event={"ID":"9ea60091-a515-422f-8cde-42894f870611","Type":"ContainerDied","Data":"d2319ed3b58bd33218ab277c07d75d92c3a4c684fc8e38aeacad6a5aaba9cf03"} Sep 06 14:26:36.530274 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.530257 2577 scope.go:117] "RemoveContainer" containerID="0e86e0b8a3a026b1fbf0f3738b0331771f9aeb85bd7a8c721d307d58f60a220c" Sep 06 14:26:36.530567 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:26:36.530538 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e86e0b8a3a026b1fbf0f3738b0331771f9aeb85bd7a8c721d307d58f60a220c\": container with ID starting with 0e86e0b8a3a026b1fbf0f3738b0331771f9aeb85bd7a8c721d307d58f60a220c not found: ID does not exist" containerID="0e86e0b8a3a026b1fbf0f3738b0331771f9aeb85bd7a8c721d307d58f60a220c" Sep 06 14:26:36.530640 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.530572 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e86e0b8a3a026b1fbf0f3738b0331771f9aeb85bd7a8c721d307d58f60a220c"} err="failed to get container status \"0e86e0b8a3a026b1fbf0f3738b0331771f9aeb85bd7a8c721d307d58f60a220c\": rpc error: code = NotFound desc = could not find container \"0e86e0b8a3a026b1fbf0f3738b0331771f9aeb85bd7a8c721d307d58f60a220c\": container with ID starting with 0e86e0b8a3a026b1fbf0f3738b0331771f9aeb85bd7a8c721d307d58f60a220c not found: ID does not exist" Sep 06 14:26:36.530640 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.530587 2577 scope.go:117] "RemoveContainer" containerID="eeb492cf7fea75b6e5407ab22cac4c8183134da2831a8d639b2765cedbf96402" Sep 06 14:26:36.538776 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.538757 2577 scope.go:117] "RemoveContainer" containerID="eeb492cf7fea75b6e5407ab22cac4c8183134da2831a8d639b2765cedbf96402" Sep 06 14:26:36.539105 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:26:36.539083 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb492cf7fea75b6e5407ab22cac4c8183134da2831a8d639b2765cedbf96402\": container with ID starting with eeb492cf7fea75b6e5407ab22cac4c8183134da2831a8d639b2765cedbf96402 not found: ID does not exist" containerID="eeb492cf7fea75b6e5407ab22cac4c8183134da2831a8d639b2765cedbf96402" Sep 06 14:26:36.539161 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.539114 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb492cf7fea75b6e5407ab22cac4c8183134da2831a8d639b2765cedbf96402"} err="failed to get container status \"eeb492cf7fea75b6e5407ab22cac4c8183134da2831a8d639b2765cedbf96402\": rpc error: code = NotFound desc = could not find container \"eeb492cf7fea75b6e5407ab22cac4c8183134da2831a8d639b2765cedbf96402\": container with ID starting with eeb492cf7fea75b6e5407ab22cac4c8183134da2831a8d639b2765cedbf96402 not found: ID does not exist" Sep 06 14:26:36.546739 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.546720 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2"] Sep 06 14:26:36.550012 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.549991 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c443d8-8495476d58-tq8d2"] Sep 06 14:26:36.559417 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.559396 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq"] Sep 06 14:26:36.562775 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:36.562753 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c443d8-767fd8848f-hq6pq"] Sep 06 14:26:37.067151 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:37.067117 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0608cdb9-87d3-4a8d-9c52-e5246d8b38e2" path="/var/lib/kubelet/pods/0608cdb9-87d3-4a8d-9c52-e5246d8b38e2/volumes" Sep 06 14:26:37.067654 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:37.067634 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ea60091-a515-422f-8cde-42894f870611" path="/var/lib/kubelet/pods/9ea60091-a515-422f-8cde-42894f870611/volumes" Sep 06 14:26:39.386763 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:39.386729 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:26:41.769503 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:41.769466 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:41.822424 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:41.822396 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:42.028784 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:42.028696 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:26:46.490988 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:26:46.490952 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:28:00.721290 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:28:00.721256 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:28:06.091206 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:28:06.091171 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:28:07.291623 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:28:07.291588 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:28:07.842209 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:28:07.842163 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:28:09.711542 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:28:09.711492 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:28:12.035402 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:28:12.035358 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:28:32.097300 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:28:32.097268 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:28:40.283883 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:28:40.283847 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:28:47.393489 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:28:47.393450 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:28:49.450400 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:28:49.450370 2577 scope.go:117] "RemoveContainer" containerID="0d505d6f3d0e90648c3d49a4b2eb216cb4e55b286b089c80905c23e4c828f5ac" Sep 06 14:29:07.284660 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:29:07.284627 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:29:14.098327 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:29:14.098291 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:30:41.585054 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:30:41.584973 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:30:50.895922 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:30:50.895872 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:30:51.174067 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:30:51.173988 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:31:11.625265 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:31:11.625227 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:31:14.197475 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:31:14.197427 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:31:45.186088 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:31:45.186052 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:31:48.198967 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:31:48.198926 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:32:20.396019 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:20.395932 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:32:22.027401 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:22.027361 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:32:22.696599 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:22.696569 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:32:31.588450 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:31.588413 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:32:38.687709 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:38.687673 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:32:45.281560 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.281520 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9"] Sep 06 14:32:45.282145 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282125 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea60091-a515-422f-8cde-42894f870611" containerName="payload-processing" Sep 06 14:32:45.282237 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282147 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea60091-a515-422f-8cde-42894f870611" containerName="payload-processing" Sep 06 14:32:45.282237 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282162 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78c162f6-4168-41ef-8c42-dce69c734264" containerName="istio-proxy" Sep 06 14:32:45.282237 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282172 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78c162f6-4168-41ef-8c42-dce69c734264" containerName="istio-proxy" Sep 06 14:32:45.282237 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282186 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7da9281-f964-4eae-a5a7-c2cab274009a" containerName="payload-pre-processing" Sep 06 14:32:45.282237 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282196 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7da9281-f964-4eae-a5a7-c2cab274009a" containerName="payload-pre-processing" Sep 06 14:32:45.282237 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282218 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78c162f6-4168-41ef-8c42-dce69c734264" containerName="istio-proxy" Sep 06 14:32:45.282237 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282226 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78c162f6-4168-41ef-8c42-dce69c734264" containerName="istio-proxy" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282245 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerName="istio-proxy" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282253 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerName="istio-proxy" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282271 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041a1de8-413c-43e6-8e8a-52ceb5f3ee4e" containerName="payload-pre-processing" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282280 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="041a1de8-413c-43e6-8e8a-52ceb5f3ee4e" containerName="payload-pre-processing" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282292 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0608cdb9-87d3-4a8d-9c52-e5246d8b38e2" containerName="payload-pre-processing" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282300 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0608cdb9-87d3-4a8d-9c52-e5246d8b38e2" containerName="payload-pre-processing" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282313 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5301ac3-0aeb-43c6-85ec-88b74c67b59c" containerName="maas-api" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282322 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5301ac3-0aeb-43c6-85ec-88b74c67b59c" containerName="maas-api" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282335 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c3bd526-2927-4d9a-83ad-b7b3c414e4ff" containerName="payload-pre-processing" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282343 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c3bd526-2927-4d9a-83ad-b7b3c414e4ff" containerName="payload-pre-processing" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282432 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78c162f6-4168-41ef-8c42-dce69c734264" containerName="istio-proxy" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282443 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0608cdb9-87d3-4a8d-9c52-e5246d8b38e2" containerName="payload-pre-processing" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282455 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7da9281-f964-4eae-a5a7-c2cab274009a" containerName="payload-pre-processing" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282466 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerName="istio-proxy" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282474 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ea60091-a515-422f-8cde-42894f870611" containerName="payload-processing" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282482 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c3bd526-2927-4d9a-83ad-b7b3c414e4ff" containerName="payload-pre-processing" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282492 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78c162f6-4168-41ef-8c42-dce69c734264" containerName="istio-proxy" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282500 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5301ac3-0aeb-43c6-85ec-88b74c67b59c" containerName="maas-api" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282511 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="041a1de8-413c-43e6-8e8a-52ceb5f3ee4e" containerName="payload-pre-processing" Sep 06 14:32:45.282601 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.282521 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerName="istio-proxy" Sep 06 14:32:45.286091 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.286069 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:45.289215 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.289192 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4lj84\"" Sep 06 14:32:45.289326 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.289217 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 06 14:32:45.289326 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.289229 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 06 14:32:45.289326 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.289250 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 06 14:32:45.289326 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.289193 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 14:32:45.295970 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.295946 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9"] Sep 06 14:32:45.313665 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.313637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc72d\" (UniqueName: \"kubernetes.io/projected/5062b241-dd3e-4fd4-9f28-c2a714066a86-kube-api-access-bc72d\") pod \"maas-api-7ff94db9d5-qtsh9\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:45.313787 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.313682 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5062b241-dd3e-4fd4-9f28-c2a714066a86-metrics-certs\") pod \"maas-api-7ff94db9d5-qtsh9\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:45.313787 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.313746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5062b241-dd3e-4fd4-9f28-c2a714066a86-openshift-service-ca\") pod \"maas-api-7ff94db9d5-qtsh9\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:45.313889 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.313802 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5062b241-dd3e-4fd4-9f28-c2a714066a86-maas-api-tls\") pod \"maas-api-7ff94db9d5-qtsh9\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:45.414527 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.414485 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc72d\" (UniqueName: \"kubernetes.io/projected/5062b241-dd3e-4fd4-9f28-c2a714066a86-kube-api-access-bc72d\") pod \"maas-api-7ff94db9d5-qtsh9\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:45.414527 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.414527 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5062b241-dd3e-4fd4-9f28-c2a714066a86-metrics-certs\") pod \"maas-api-7ff94db9d5-qtsh9\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:45.414769 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.414572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5062b241-dd3e-4fd4-9f28-c2a714066a86-openshift-service-ca\") pod \"maas-api-7ff94db9d5-qtsh9\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:45.414769 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.414616 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5062b241-dd3e-4fd4-9f28-c2a714066a86-maas-api-tls\") pod \"maas-api-7ff94db9d5-qtsh9\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:45.415219 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.415197 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5062b241-dd3e-4fd4-9f28-c2a714066a86-openshift-service-ca\") pod \"maas-api-7ff94db9d5-qtsh9\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:45.416881 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.416861 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5062b241-dd3e-4fd4-9f28-c2a714066a86-maas-api-tls\") pod \"maas-api-7ff94db9d5-qtsh9\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:45.416985 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.416949 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5062b241-dd3e-4fd4-9f28-c2a714066a86-metrics-certs\") pod \"maas-api-7ff94db9d5-qtsh9\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:45.421698 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.421675 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc72d\" (UniqueName: \"kubernetes.io/projected/5062b241-dd3e-4fd4-9f28-c2a714066a86-kube-api-access-bc72d\") pod \"maas-api-7ff94db9d5-qtsh9\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:45.598216 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.598145 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:45.868019 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.867993 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9"] Sep 06 14:32:45.870357 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:32:45.870330 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5062b241_dd3e_4fd4_9f28_c2a714066a86.slice/crio-ba37fc3f262c61a2cd9a1aedbf1658276f698c018ce2b045a8625a37255bdfd5 WatchSource:0}: Error finding container ba37fc3f262c61a2cd9a1aedbf1658276f698c018ce2b045a8625a37255bdfd5: Status 404 returned error can't find the container with id ba37fc3f262c61a2cd9a1aedbf1658276f698c018ce2b045a8625a37255bdfd5 Sep 06 14:32:45.871643 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:45.871624 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 14:32:46.088133 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:46.088095 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:32:46.318555 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:46.318519 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" event={"ID":"5062b241-dd3e-4fd4-9f28-c2a714066a86","Type":"ContainerStarted","Data":"ba37fc3f262c61a2cd9a1aedbf1658276f698c018ce2b045a8625a37255bdfd5"} Sep 06 14:32:47.326948 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:47.326902 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" event={"ID":"5062b241-dd3e-4fd4-9f28-c2a714066a86","Type":"ContainerStarted","Data":"71b8c12487226ef0c6cebb2cafc130cfef300be286c11add8ecbc2b86059be95"} Sep 06 14:32:47.327335 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:47.327046 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:47.343857 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:47.343803 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" podStartSLOduration=1.942494569 podStartE2EDuration="2.343788293s" totalImagesPullingTime="401.293724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:32:47.340813519 +0000 UTC m=+2098.796302381" watchObservedRunningTime="2026-09-06 14:32:47.343788293 +0000 UTC m=+2098.799277154" Sep 06 14:32:53.103104 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.103066 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9"] Sep 06 14:32:53.103574 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.103282 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" podUID="5062b241-dd3e-4fd4-9f28-c2a714066a86" containerName="maas-api" containerID="cri-o://71b8c12487226ef0c6cebb2cafc130cfef300be286c11add8ecbc2b86059be95" gracePeriod=30 Sep 06 14:32:53.108461 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.108435 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:53.132768 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.132743 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx"] Sep 06 14:32:53.133194 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.133177 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerName="istio-proxy" Sep 06 14:32:53.133277 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.133196 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c321624-5acb-4712-ba3b-20d5450b1fe6" containerName="istio-proxy" Sep 06 14:32:53.136330 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.136311 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:32:53.146092 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.146033 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx"] Sep 06 14:32:53.180922 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.180885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bac10b-930f-40f9-a0bf-b131baad1ea3-openshift-service-ca\") pod \"maas-api-6669d75c8f-xsjfx\" (UID: \"96bac10b-930f-40f9-a0bf-b131baad1ea3\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:32:53.181023 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.180950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bac10b-930f-40f9-a0bf-b131baad1ea3-maas-api-tls\") pod \"maas-api-6669d75c8f-xsjfx\" (UID: \"96bac10b-930f-40f9-a0bf-b131baad1ea3\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:32:53.181023 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.181001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dscrk\" (UniqueName: \"kubernetes.io/projected/96bac10b-930f-40f9-a0bf-b131baad1ea3-kube-api-access-dscrk\") pod \"maas-api-6669d75c8f-xsjfx\" (UID: \"96bac10b-930f-40f9-a0bf-b131baad1ea3\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:32:53.181187 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.181167 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bac10b-930f-40f9-a0bf-b131baad1ea3-metrics-certs\") pod \"maas-api-6669d75c8f-xsjfx\" (UID: \"96bac10b-930f-40f9-a0bf-b131baad1ea3\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:32:53.189756 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.189732 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:32:53.282183 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.282155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bac10b-930f-40f9-a0bf-b131baad1ea3-openshift-service-ca\") pod \"maas-api-6669d75c8f-xsjfx\" (UID: \"96bac10b-930f-40f9-a0bf-b131baad1ea3\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:32:53.282338 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.282214 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bac10b-930f-40f9-a0bf-b131baad1ea3-maas-api-tls\") pod \"maas-api-6669d75c8f-xsjfx\" (UID: \"96bac10b-930f-40f9-a0bf-b131baad1ea3\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:32:53.282338 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.282282 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dscrk\" (UniqueName: \"kubernetes.io/projected/96bac10b-930f-40f9-a0bf-b131baad1ea3-kube-api-access-dscrk\") pod \"maas-api-6669d75c8f-xsjfx\" (UID: \"96bac10b-930f-40f9-a0bf-b131baad1ea3\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:32:53.282779 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.282743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bac10b-930f-40f9-a0bf-b131baad1ea3-metrics-certs\") pod \"maas-api-6669d75c8f-xsjfx\" (UID: \"96bac10b-930f-40f9-a0bf-b131baad1ea3\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:32:53.282981 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.282955 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bac10b-930f-40f9-a0bf-b131baad1ea3-openshift-service-ca\") pod \"maas-api-6669d75c8f-xsjfx\" (UID: \"96bac10b-930f-40f9-a0bf-b131baad1ea3\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:32:53.285306 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.285280 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bac10b-930f-40f9-a0bf-b131baad1ea3-metrics-certs\") pod \"maas-api-6669d75c8f-xsjfx\" (UID: \"96bac10b-930f-40f9-a0bf-b131baad1ea3\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:32:53.285727 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.285708 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bac10b-930f-40f9-a0bf-b131baad1ea3-maas-api-tls\") pod \"maas-api-6669d75c8f-xsjfx\" (UID: \"96bac10b-930f-40f9-a0bf-b131baad1ea3\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:32:53.289900 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.289874 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dscrk\" (UniqueName: \"kubernetes.io/projected/96bac10b-930f-40f9-a0bf-b131baad1ea3-kube-api-access-dscrk\") pod \"maas-api-6669d75c8f-xsjfx\" (UID: \"96bac10b-930f-40f9-a0bf-b131baad1ea3\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:32:53.341343 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.341320 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:53.360460 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.360370 2577 generic.go:356] "Generic (PLEG): container finished" podID="5062b241-dd3e-4fd4-9f28-c2a714066a86" containerID="71b8c12487226ef0c6cebb2cafc130cfef300be286c11add8ecbc2b86059be95" exitCode=0 Sep 06 14:32:53.360460 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.360409 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" event={"ID":"5062b241-dd3e-4fd4-9f28-c2a714066a86","Type":"ContainerDied","Data":"71b8c12487226ef0c6cebb2cafc130cfef300be286c11add8ecbc2b86059be95"} Sep 06 14:32:53.360621 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.360479 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" Sep 06 14:32:53.360621 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.360500 2577 scope.go:117] "RemoveContainer" containerID="71b8c12487226ef0c6cebb2cafc130cfef300be286c11add8ecbc2b86059be95" Sep 06 14:32:53.360621 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.360486 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9" event={"ID":"5062b241-dd3e-4fd4-9f28-c2a714066a86","Type":"ContainerDied","Data":"ba37fc3f262c61a2cd9a1aedbf1658276f698c018ce2b045a8625a37255bdfd5"} Sep 06 14:32:53.370011 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.369990 2577 scope.go:117] "RemoveContainer" containerID="71b8c12487226ef0c6cebb2cafc130cfef300be286c11add8ecbc2b86059be95" Sep 06 14:32:53.370284 ip-10-0-132-113 kubenswrapper[2577]: E0906 14:32:53.370261 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b8c12487226ef0c6cebb2cafc130cfef300be286c11add8ecbc2b86059be95\": container with ID starting with 71b8c12487226ef0c6cebb2cafc130cfef300be286c11add8ecbc2b86059be95 not found: ID does not exist" containerID="71b8c12487226ef0c6cebb2cafc130cfef300be286c11add8ecbc2b86059be95" Sep 06 14:32:53.370349 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.370299 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b8c12487226ef0c6cebb2cafc130cfef300be286c11add8ecbc2b86059be95"} err="failed to get container status \"71b8c12487226ef0c6cebb2cafc130cfef300be286c11add8ecbc2b86059be95\": rpc error: code = NotFound desc = could not find container \"71b8c12487226ef0c6cebb2cafc130cfef300be286c11add8ecbc2b86059be95\": container with ID starting with 71b8c12487226ef0c6cebb2cafc130cfef300be286c11add8ecbc2b86059be95 not found: ID does not exist" Sep 06 14:32:53.383575 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.383555 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5062b241-dd3e-4fd4-9f28-c2a714066a86-maas-api-tls\") pod \"5062b241-dd3e-4fd4-9f28-c2a714066a86\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " Sep 06 14:32:53.383664 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.383590 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5062b241-dd3e-4fd4-9f28-c2a714066a86-metrics-certs\") pod \"5062b241-dd3e-4fd4-9f28-c2a714066a86\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " Sep 06 14:32:53.383664 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.383615 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5062b241-dd3e-4fd4-9f28-c2a714066a86-openshift-service-ca\") pod \"5062b241-dd3e-4fd4-9f28-c2a714066a86\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " Sep 06 14:32:53.383664 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.383651 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc72d\" (UniqueName: \"kubernetes.io/projected/5062b241-dd3e-4fd4-9f28-c2a714066a86-kube-api-access-bc72d\") pod \"5062b241-dd3e-4fd4-9f28-c2a714066a86\" (UID: \"5062b241-dd3e-4fd4-9f28-c2a714066a86\") " Sep 06 14:32:53.384007 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.383978 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5062b241-dd3e-4fd4-9f28-c2a714066a86-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5062b241-dd3e-4fd4-9f28-c2a714066a86" (UID: "5062b241-dd3e-4fd4-9f28-c2a714066a86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:32:53.385639 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.385611 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5062b241-dd3e-4fd4-9f28-c2a714066a86-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5062b241-dd3e-4fd4-9f28-c2a714066a86" (UID: "5062b241-dd3e-4fd4-9f28-c2a714066a86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:32:53.385639 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.385631 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5062b241-dd3e-4fd4-9f28-c2a714066a86-kube-api-access-bc72d" (OuterVolumeSpecName: "kube-api-access-bc72d") pod "5062b241-dd3e-4fd4-9f28-c2a714066a86" (UID: "5062b241-dd3e-4fd4-9f28-c2a714066a86"). InnerVolumeSpecName "kube-api-access-bc72d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:32:53.385753 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.385643 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5062b241-dd3e-4fd4-9f28-c2a714066a86-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5062b241-dd3e-4fd4-9f28-c2a714066a86" (UID: "5062b241-dd3e-4fd4-9f28-c2a714066a86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:32:53.481289 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.481255 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:32:53.484852 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.484824 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5062b241-dd3e-4fd4-9f28-c2a714066a86-maas-api-tls\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:32:53.484852 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.484848 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5062b241-dd3e-4fd4-9f28-c2a714066a86-metrics-certs\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:32:53.484852 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.484857 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5062b241-dd3e-4fd4-9f28-c2a714066a86-openshift-service-ca\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:32:53.485076 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.484866 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc72d\" (UniqueName: \"kubernetes.io/projected/5062b241-dd3e-4fd4-9f28-c2a714066a86-kube-api-access-bc72d\") on node \"ip-10-0-132-113.ec2.internal\" DevicePath \"\"" Sep 06 14:32:53.680579 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.680549 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9"] Sep 06 14:32:53.686779 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.686745 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff94db9d5-qtsh9"] Sep 06 14:32:53.751394 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:53.751363 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx"] Sep 06 14:32:53.753432 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:32:53.753409 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96bac10b_930f_40f9_a0bf_b131baad1ea3.slice/crio-149cb85a1c74cdf6c6ad1ce5776047829b18c5bfb37c9c54563cc16c3d004c6c WatchSource:0}: Error finding container 149cb85a1c74cdf6c6ad1ce5776047829b18c5bfb37c9c54563cc16c3d004c6c: Status 404 returned error can't find the container with id 149cb85a1c74cdf6c6ad1ce5776047829b18c5bfb37c9c54563cc16c3d004c6c Sep 06 14:32:54.370391 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:54.370301 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" event={"ID":"96bac10b-930f-40f9-a0bf-b131baad1ea3","Type":"ContainerStarted","Data":"a541b25501f98fc6ae86cb9ff2d29e366b263e622214170d65daa0d2be8e9eb2"} Sep 06 14:32:54.370391 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:54.370365 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" event={"ID":"96bac10b-930f-40f9-a0bf-b131baad1ea3","Type":"ContainerStarted","Data":"149cb85a1c74cdf6c6ad1ce5776047829b18c5bfb37c9c54563cc16c3d004c6c"} Sep 06 14:32:54.370838 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:54.370436 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:32:54.384999 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:54.384944 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" podStartSLOduration=1.063139741 podStartE2EDuration="1.384896732s" totalImagesPullingTime="321.756991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:32:54.383743145 +0000 UTC m=+2105.839232007" watchObservedRunningTime="2026-09-06 14:32:54.384896732 +0000 UTC m=+2105.840385596" Sep 06 14:32:55.067142 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:32:55.067109 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5062b241-dd3e-4fd4-9f28-c2a714066a86" path="/var/lib/kubelet/pods/5062b241-dd3e-4fd4-9f28-c2a714066a86/volumes" Sep 06 14:33:00.381301 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:00.381274 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6669d75c8f-xsjfx" Sep 06 14:33:02.687002 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:02.686971 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:33:09.892067 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:09.892031 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:33:18.690409 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:18.690373 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7jbpf"] Sep 06 14:33:26.383496 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:26.383463 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-cwsj8_a4d5ba38-49cd-4124-a20d-fbb5a0d27506/manager/0.log" Sep 06 14:33:26.720050 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:26.719942 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6858f974-zgl8s_a4f783ae-85c9-4ab0-aee3-ca3573820243/copy-manifests/0.log" Sep 06 14:33:26.737086 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:26.737051 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6858f974-zgl8s_a4f783ae-85c9-4ab0-aee3-ca3573820243/manager/0.log" Sep 06 14:33:27.156988 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:27.156955 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6669d75c8f-xsjfx_96bac10b-930f-40f9-a0bf-b131baad1ea3/maas-api/0.log" Sep 06 14:33:28.773819 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:28.773791 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-7jbpf_163b006d-dcb7-4c0b-acfd-579d88d2b6fb/limitador/0.log" Sep 06 14:33:29.349553 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:29.349516 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wfwsh_42eb9483-2a9e-4816-957c-5bfedefda1d2/discovery/0.log" Sep 06 14:33:29.671757 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:29.671675 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9m4jl_0479d3c1-0872-435a-8133-8e570afa1d85/istio-proxy/1.log" Sep 06 14:33:29.897921 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:29.897888 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-shz4m_434bf1f2-d6df-4fc0-a8ae-49216214dd8c/payload-processing/0.log" Sep 06 14:33:30.768027 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:30.767994 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdzkh2_b0766686-d24e-46e6-b3a5-0f25393d0fc4/main/0.log" Sep 06 14:33:30.874740 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:30.874702 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-mpw88_02261a3a-c803-4c8d-a93d-db27f1543676/main/0.log" Sep 06 14:33:36.666314 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:36.666269 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:33:36.767323 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:36.767283 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:33:39.450208 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:39.450177 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9fqd2_53540329-a02b-4fcb-ac80-8d6bf3489b10/global-pull-secret-syncer/0.log" Sep 06 14:33:39.572164 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:39.572129 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" Sep 06 14:33:39.656158 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:39.656126 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-113.ec2.internal_d4e8e4996366f70a783b63f04db7c684/haproxy/0.log" Sep 06 14:33:40.610866 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:40.610830 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:33:40.712279 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:40.712249 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ms5fd_ab2f34a9-090c-4b27-a511-310e6d0e50ed/konnectivity-agent/0.log" limit=1024 Sep 06 14:33:44.628187 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:44.628160 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-7jbpf_163b006d-dcb7-4c0b-acfd-579d88d2b6fb/limitador/0.log" Sep 06 14:33:46.078396 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.078365 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f230577-b8d8-4b16-92e3-a6b51927611f/alertmanager/0.log" Sep 06 14:33:46.095989 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.095946 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f230577-b8d8-4b16-92e3-a6b51927611f/config-reloader/0.log" Sep 06 14:33:46.112613 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.112594 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f230577-b8d8-4b16-92e3-a6b51927611f/kube-rbac-proxy-web/0.log" Sep 06 14:33:46.132045 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.132021 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f230577-b8d8-4b16-92e3-a6b51927611f/kube-rbac-proxy/0.log" Sep 06 14:33:46.152091 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.152065 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f230577-b8d8-4b16-92e3-a6b51927611f/kube-rbac-proxy-metric/0.log" Sep 06 14:33:46.168725 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.168697 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f230577-b8d8-4b16-92e3-a6b51927611f/prom-label-proxy/0.log" Sep 06 14:33:46.188715 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.188693 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f230577-b8d8-4b16-92e3-a6b51927611f/init-config-reloader/0.log" Sep 06 14:33:46.252631 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.252604 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-7qp65_3ec859a2-4603-4f11-809d-83112426a10d/kube-state-metrics/0.log" Sep 06 14:33:46.265981 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.265954 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-7qp65_3ec859a2-4603-4f11-809d-83112426a10d/kube-rbac-proxy-main/0.log" Sep 06 14:33:46.283307 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.283281 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-7qp65_3ec859a2-4603-4f11-809d-83112426a10d/kube-rbac-proxy-self/0.log" Sep 06 14:33:46.306879 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.306822 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6996bf4777-kh87z_e080444a-1143-4e22-bcbe-b2180243a549/metrics-server/0.log" Sep 06 14:33:46.331416 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.331395 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-hjbv5_3b4f0419-be6f-4900-8116-6fdbe7278997/monitoring-plugin/0.log" Sep 06 14:33:46.368772 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.368747 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9wsfj_b157e249-7489-4dd2-bd2b-47f85605cd9d/node-exporter/0.log" Sep 06 14:33:46.387102 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.387080 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9wsfj_b157e249-7489-4dd2-bd2b-47f85605cd9d/kube-rbac-proxy/0.log" Sep 06 14:33:46.407472 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.407450 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9wsfj_b157e249-7489-4dd2-bd2b-47f85605cd9d/init-textfile/0.log" Sep 06 14:33:46.596002 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.595933 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-jfkwt_9f2faa94-8b19-46ae-9c4b-0fb3b8de760a/kube-rbac-proxy-main/0.log" Sep 06 14:33:46.625023 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.624998 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-jfkwt_9f2faa94-8b19-46ae-9c4b-0fb3b8de760a/kube-rbac-proxy-self/0.log" Sep 06 14:33:46.649780 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.649755 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-jfkwt_9f2faa94-8b19-46ae-9c4b-0fb3b8de760a/openshift-state-metrics/0.log" Sep 06 14:33:46.699184 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.699157 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c1307f0-e605-4d79-9f62-dda3c55b0881/prometheus/0.log" Sep 06 14:33:46.724895 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.724872 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c1307f0-e605-4d79-9f62-dda3c55b0881/config-reloader/0.log" Sep 06 14:33:46.760552 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.760522 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c1307f0-e605-4d79-9f62-dda3c55b0881/thanos-sidecar/0.log" Sep 06 14:33:46.780257 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.780233 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c1307f0-e605-4d79-9f62-dda3c55b0881/kube-rbac-proxy-web/0.log" Sep 06 14:33:46.809736 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.809714 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c1307f0-e605-4d79-9f62-dda3c55b0881/kube-rbac-proxy/0.log" Sep 06 14:33:46.832226 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.832204 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c1307f0-e605-4d79-9f62-dda3c55b0881/kube-rbac-proxy-thanos/0.log" Sep 06 14:33:46.854134 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.854075 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c1307f0-e605-4d79-9f62-dda3c55b0881/init-config-reloader/0.log" Sep 06 14:33:46.880318 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.880295 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-55d5f_9a6639dd-bc29-4b35-8b48-595283323be3/prometheus-operator/0.log" Sep 06 14:33:46.897389 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.897372 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-55d5f_9a6639dd-bc29-4b35-8b48-595283323be3/kube-rbac-proxy/0.log" Sep 06 14:33:46.919305 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.919285 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-86c2s_552fa68e-cd7a-473c-a685-b49178b6686b/prometheus-operator-admission-webhook/0.log" Sep 06 14:33:46.967963 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.967942 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-567955b447-sm9q6_c8b26446-fc0a-4422-b6fd-5771b7e9a14a/telemeter-client/0.log" Sep 06 14:33:46.986877 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:46.986855 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-567955b447-sm9q6_c8b26446-fc0a-4422-b6fd-5771b7e9a14a/reload/0.log" Sep 06 14:33:47.004876 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:47.004860 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-567955b447-sm9q6_c8b26446-fc0a-4422-b6fd-5771b7e9a14a/kube-rbac-proxy/0.log" Sep 06 14:33:47.034229 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:47.034208 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7876d5bf99-mn6rr_9eb9bded-0cf1-41ab-b338-87538c1644a6/thanos-query/0.log" Sep 06 14:33:47.051173 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:47.051154 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7876d5bf99-mn6rr_9eb9bded-0cf1-41ab-b338-87538c1644a6/kube-rbac-proxy-web/0.log" Sep 06 14:33:47.068610 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:47.068585 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7876d5bf99-mn6rr_9eb9bded-0cf1-41ab-b338-87538c1644a6/kube-rbac-proxy/0.log" Sep 06 14:33:47.085383 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:47.085366 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7876d5bf99-mn6rr_9eb9bded-0cf1-41ab-b338-87538c1644a6/prom-label-proxy/0.log" Sep 06 14:33:47.102044 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:47.102022 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7876d5bf99-mn6rr_9eb9bded-0cf1-41ab-b338-87538c1644a6/kube-rbac-proxy-rules/0.log" Sep 06 14:33:47.118144 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:47.118080 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7876d5bf99-mn6rr_9eb9bded-0cf1-41ab-b338-87538c1644a6/kube-rbac-proxy-metrics/0.log" Sep 06 14:33:49.716056 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.716026 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-4h9mc_c7c35f62-5d92-4fd0-82b4-cc6274ecd0c1/download-server/0.log" Sep 06 14:33:49.759511 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.759435 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv"] Sep 06 14:33:49.759955 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.759921 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5062b241-dd3e-4fd4-9f28-c2a714066a86" containerName="maas-api" Sep 06 14:33:49.759955 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.759947 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5062b241-dd3e-4fd4-9f28-c2a714066a86" containerName="maas-api" Sep 06 14:33:49.760172 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.760056 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5062b241-dd3e-4fd4-9f28-c2a714066a86" containerName="maas-api" Sep 06 14:33:49.764688 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.764671 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:49.766819 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.766801 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfljt\"/\"openshift-service-ca.crt\"" Sep 06 14:33:49.766926 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.766863 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfljt\"/\"kube-root-ca.crt\"" Sep 06 14:33:49.767575 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.767556 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pfljt\"/\"default-dockercfg-9w4wc\"" Sep 06 14:33:49.770797 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.770776 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv"] Sep 06 14:33:49.897975 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.897941 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/084a82ad-eacb-450f-90c5-664df76627ab-lib-modules\") pod \"perf-node-gather-daemonset-7smzv\" (UID: \"084a82ad-eacb-450f-90c5-664df76627ab\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:49.898154 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.897984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/084a82ad-eacb-450f-90c5-664df76627ab-sys\") pod \"perf-node-gather-daemonset-7smzv\" (UID: \"084a82ad-eacb-450f-90c5-664df76627ab\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:49.898154 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.898042 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/084a82ad-eacb-450f-90c5-664df76627ab-proc\") pod \"perf-node-gather-daemonset-7smzv\" (UID: \"084a82ad-eacb-450f-90c5-664df76627ab\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:49.898154 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.898066 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/084a82ad-eacb-450f-90c5-664df76627ab-podres\") pod \"perf-node-gather-daemonset-7smzv\" (UID: \"084a82ad-eacb-450f-90c5-664df76627ab\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:49.898154 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.898118 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqvhc\" (UniqueName: \"kubernetes.io/projected/084a82ad-eacb-450f-90c5-664df76627ab-kube-api-access-fqvhc\") pod \"perf-node-gather-daemonset-7smzv\" (UID: \"084a82ad-eacb-450f-90c5-664df76627ab\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:49.998930 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.998873 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/084a82ad-eacb-450f-90c5-664df76627ab-lib-modules\") pod \"perf-node-gather-daemonset-7smzv\" (UID: \"084a82ad-eacb-450f-90c5-664df76627ab\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:49.999116 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.998961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/084a82ad-eacb-450f-90c5-664df76627ab-sys\") pod \"perf-node-gather-daemonset-7smzv\" (UID: \"084a82ad-eacb-450f-90c5-664df76627ab\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:49.999116 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.999008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/084a82ad-eacb-450f-90c5-664df76627ab-proc\") pod \"perf-node-gather-daemonset-7smzv\" (UID: \"084a82ad-eacb-450f-90c5-664df76627ab\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:49.999116 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.999029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/084a82ad-eacb-450f-90c5-664df76627ab-podres\") pod \"perf-node-gather-daemonset-7smzv\" (UID: \"084a82ad-eacb-450f-90c5-664df76627ab\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:49.999116 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.999071 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/084a82ad-eacb-450f-90c5-664df76627ab-lib-modules\") pod \"perf-node-gather-daemonset-7smzv\" (UID: \"084a82ad-eacb-450f-90c5-664df76627ab\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:49.999116 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.999084 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/084a82ad-eacb-450f-90c5-664df76627ab-proc\") pod \"perf-node-gather-daemonset-7smzv\" (UID: \"084a82ad-eacb-450f-90c5-664df76627ab\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:49.999116 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.999071 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/084a82ad-eacb-450f-90c5-664df76627ab-sys\") pod \"perf-node-gather-daemonset-7smzv\" (UID: \"084a82ad-eacb-450f-90c5-664df76627ab\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:49.999116 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.999077 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqvhc\" (UniqueName: \"kubernetes.io/projected/084a82ad-eacb-450f-90c5-664df76627ab-kube-api-access-fqvhc\") pod \"perf-node-gather-daemonset-7smzv\" (UID: \"084a82ad-eacb-450f-90c5-664df76627ab\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:49.999489 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:49.999149 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/084a82ad-eacb-450f-90c5-664df76627ab-podres\") pod \"perf-node-gather-daemonset-7smzv\" (UID: \"084a82ad-eacb-450f-90c5-664df76627ab\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:50.006411 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:50.006387 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqvhc\" (UniqueName: \"kubernetes.io/projected/084a82ad-eacb-450f-90c5-664df76627ab-kube-api-access-fqvhc\") pod \"perf-node-gather-daemonset-7smzv\" (UID: \"084a82ad-eacb-450f-90c5-664df76627ab\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:50.075932 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:50.075896 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:50.343839 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:50.343811 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv"] Sep 06 14:33:50.346464 ip-10-0-132-113 kubenswrapper[2577]: W0906 14:33:50.346439 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod084a82ad_eacb_450f_90c5_664df76627ab.slice/crio-6b93c6b367ea6b54bf67019a17e7485b393f0e337d961505ceffd6f5e3406815 WatchSource:0}: Error finding container 6b93c6b367ea6b54bf67019a17e7485b393f0e337d961505ceffd6f5e3406815: Status 404 returned error can't find the container with id 6b93c6b367ea6b54bf67019a17e7485b393f0e337d961505ceffd6f5e3406815 Sep 06 14:33:50.660018 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:50.659937 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" event={"ID":"084a82ad-eacb-450f-90c5-664df76627ab","Type":"ContainerStarted","Data":"a6ac6fbbb0f702a40f82e54532d3e438f8fda68ff7b240f977fb6c5ede0a66d3"} Sep 06 14:33:50.660018 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:50.660015 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" event={"ID":"084a82ad-eacb-450f-90c5-664df76627ab","Type":"ContainerStarted","Data":"6b93c6b367ea6b54bf67019a17e7485b393f0e337d961505ceffd6f5e3406815"} Sep 06 14:33:50.660201 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:50.660073 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:50.674830 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:50.674788 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" podStartSLOduration=1.674776525 podStartE2EDuration="1.674776525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 14:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:33:50.673885639 +0000 UTC m=+2162.129374502" watchObservedRunningTime="2026-09-06 14:33:50.674776525 +0000 UTC m=+2162.130265410" Sep 06 14:33:51.609955 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:51.609927 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lj66p_49733091-cbf3-4000-a5c2-388553158a1b/dns/0.log" Sep 06 14:33:51.626887 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:51.626864 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lj66p_49733091-cbf3-4000-a5c2-388553158a1b/kube-rbac-proxy/0.log" Sep 06 14:33:51.684531 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:51.684501 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-42dlx_61206fdd-09e3-449e-b320-c62e226616e0/dns-node-resolver/0.log" Sep 06 14:33:52.271440 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:52.271411 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-98p47_82c12af7-6d5b-4f10-871c-442eea180684/node-ca/0.log" Sep 06 14:33:53.739061 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:53.739034 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wfwsh_42eb9483-2a9e-4816-957c-5bfedefda1d2/discovery/0.log" Sep 06 14:33:53.803972 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:53.803942 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9m4jl_0479d3c1-0872-435a-8133-8e570afa1d85/istio-proxy/0.log" Sep 06 14:33:53.821001 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:53.820974 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9m4jl_0479d3c1-0872-435a-8133-8e570afa1d85/istio-proxy/1.log" Sep 06 14:33:53.897502 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:53.897469 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-shz4m_434bf1f2-d6df-4fc0-a8ae-49216214dd8c/payload-processing/0.log" Sep 06 14:33:54.760178 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:54.760153 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kxgfz_0865f16e-e175-49f7-a333-8a59f3380eb7/serve-healthcheck-canary/0.log" Sep 06 14:33:55.322742 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:55.322713 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cq6wh_791ee34c-1b29-496a-b728-f253cd243af0/kube-rbac-proxy/0.log" Sep 06 14:33:55.340101 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:55.340077 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cq6wh_791ee34c-1b29-496a-b728-f253cd243af0/exporter/0.log" Sep 06 14:33:55.361548 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:55.361522 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cq6wh_791ee34c-1b29-496a-b728-f253cd243af0/extractor/0.log" Sep 06 14:33:56.675091 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:56.675064 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-7smzv" Sep 06 14:33:57.822500 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:57.822458 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-cwsj8_a4d5ba38-49cd-4124-a20d-fbb5a0d27506/manager/0.log" Sep 06 14:33:58.017940 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:58.017892 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6858f974-zgl8s_a4f783ae-85c9-4ab0-aee3-ca3573820243/manager/0.log" Sep 06 14:33:58.032243 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:58.032222 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6858f974-zgl8s_a4f783ae-85c9-4ab0-aee3-ca3573820243/copy-manifests/0.log" Sep 06 14:33:59.456262 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:33:59.456191 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-849f85686b-79g95_ed50ed7c-3871-439c-9d3e-e07b609fefaa/manager/0.log" Sep 06 14:34:00.692595 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:00.692565 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wfwsh_42eb9483-2a9e-4816-957c-5bfedefda1d2/discovery/0.log" Sep 06 14:34:00.741642 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:00.741613 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9m4jl_0479d3c1-0872-435a-8133-8e570afa1d85/istio-proxy/0.log" Sep 06 14:34:00.760641 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:00.760616 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9m4jl_0479d3c1-0872-435a-8133-8e570afa1d85/istio-proxy/1.log" Sep 06 14:34:00.819958 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:00.819933 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-shz4m_434bf1f2-d6df-4fc0-a8ae-49216214dd8c/payload-processing/0.log" Sep 06 14:34:02.918831 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:02.918792 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wfwsh_42eb9483-2a9e-4816-957c-5bfedefda1d2/discovery/0.log" Sep 06 14:34:02.973190 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:02.973156 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9m4jl_0479d3c1-0872-435a-8133-8e570afa1d85/istio-proxy/0.log" Sep 06 14:34:02.991031 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:02.991006 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9m4jl_0479d3c1-0872-435a-8133-8e570afa1d85/istio-proxy/1.log" Sep 06 14:34:03.057434 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:03.057409 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-shz4m_434bf1f2-d6df-4fc0-a8ae-49216214dd8c/payload-processing/0.log" Sep 06 14:34:05.767779 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:05.767746 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7x44l_cdb2dbef-18c2-4a14-a421-f15c0286c411/kube-multus-additional-cni-plugins/0.log" Sep 06 14:34:05.791095 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:05.791066 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7x44l_cdb2dbef-18c2-4a14-a421-f15c0286c411/egress-router-binary-copy/0.log" Sep 06 14:34:05.808300 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:05.808274 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7x44l_cdb2dbef-18c2-4a14-a421-f15c0286c411/cni-plugins/0.log" Sep 06 14:34:05.825278 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:05.825250 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7x44l_cdb2dbef-18c2-4a14-a421-f15c0286c411/bond-cni-plugin/0.log" Sep 06 14:34:05.842839 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:05.842772 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7x44l_cdb2dbef-18c2-4a14-a421-f15c0286c411/routeoverride-cni/0.log" Sep 06 14:34:05.862753 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:05.862729 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7x44l_cdb2dbef-18c2-4a14-a421-f15c0286c411/whereabouts-cni-bincopy/0.log" Sep 06 14:34:05.878552 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:05.878524 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7x44l_cdb2dbef-18c2-4a14-a421-f15c0286c411/whereabouts-cni/0.log" Sep 06 14:34:06.186362 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:06.186281 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ddhdd_c264c419-5ce3-43d4-b604-8580a52f3518/kube-multus/0.log" Sep 06 14:34:06.296389 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:06.296356 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bkxp5_a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd/network-metrics-daemon/0.log" Sep 06 14:34:06.308855 ip-10-0-132-113 kubenswrapper[2577]: I0906 14:34:06.308835 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bkxp5_a8d5497c-1b54-4bb9-8a57-aa2ee736e7fd/kube-rbac-proxy/0.log"