Sep 08 10:01:30.098931 ip-10-0-132-206 systemd[1]: Starting Kubernetes Kubelet... Sep 08 10:01:30.487827 ip-10-0-132-206 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 10:01:30.487827 ip-10-0-132-206 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 08 10:01:30.487827 ip-10-0-132-206 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 10:01:30.487827 ip-10-0-132-206 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 08 10:01:30.487827 ip-10-0-132-206 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 10:01:30.489060 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.488973 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 08 10:01:30.493479 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493464 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 10:01:30.493479 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493479 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493484 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493488 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493491 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493494 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493497 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493500 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493502 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493505 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493508 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493512 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493516 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493519 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493522 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493525 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493527 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493530 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493538 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493541 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 10:01:30.493545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493544 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493547 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493550 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493553 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493556 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493559 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493562 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493564 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493567 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493570 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493573 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493576 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493579 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493582 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493585 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493588 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493590 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493593 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493596 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493599 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 10:01:30.494015 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493601 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493605 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493608 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493611 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493625 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493629 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493631 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493634 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493637 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493640 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493644 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493646 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493649 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493651 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493654 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493659 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493663 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493666 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493669 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 10:01:30.494497 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493672 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493674 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493677 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493679 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493682 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493685 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493687 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493690 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493692 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493695 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493697 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493700 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493703 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493706 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493709 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493711 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493713 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493716 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493718 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493721 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493723 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 10:01:30.495008 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493726 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493729 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493732 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493735 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493737 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493740 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493743 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493747 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493750 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493752 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493755 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493758 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493762 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493765 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493767 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493770 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493773 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493775 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.493778 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494467 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 10:01:30.495529 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494473 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494476 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494479 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494481 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494484 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494487 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494490 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494493 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494496 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494499 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494502 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494504 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494507 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494510 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494513 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494516 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494518 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494521 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494524 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494526 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 10:01:30.496021 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494529 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494531 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494534 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494537 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494539 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494542 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494544 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494547 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494550 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494552 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494555 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494557 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494560 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494562 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494565 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494567 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494570 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494575 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494578 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494581 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 10:01:30.496525 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494585 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494588 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494591 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494593 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494596 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494599 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494601 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494604 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494606 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494609 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494612 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494628 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494631 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494634 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494636 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494641 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494644 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494647 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494650 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 10:01:30.497030 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494653 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494656 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494659 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494662 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494664 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494667 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494670 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494672 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494675 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494677 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494680 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494683 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494686 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494688 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494692 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494695 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494697 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494700 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494703 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494706 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 10:01:30.497498 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494708 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494710 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494713 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494716 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494718 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494721 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494723 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494726 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494729 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494732 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494735 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494737 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494741 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494745 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494748 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494751 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494754 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494756 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.494759 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 10:01:30.498014 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495154 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495159 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495162 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495165 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495169 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495172 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495174 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495177 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495180 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495183 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495186 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495190 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495193 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495196 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495198 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495201 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495204 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495207 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495209 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495212 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 08 10:01:30.498473 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495215 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495217 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495220 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495222 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495225 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495228 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495230 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495241 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495244 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495247 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495250 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495252 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495255 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495258 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495260 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495263 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495265 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495276 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495280 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 10:01:30.499001 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495283 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495286 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495289 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495291 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495294 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495297 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495300 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495302 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495305 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495307 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495310 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495313 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495315 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495318 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495320 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495323 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495325 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495328 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495330 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495333 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 10:01:30.499476 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495335 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495338 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495340 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495342 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495345 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495348 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495350 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495353 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495355 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495358 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495360 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495368 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495371 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495373 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495376 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495378 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495381 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495383 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495386 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495388 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 10:01:30.500012 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495391 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495393 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495396 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495398 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495401 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495403 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495405 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495409 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495412 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495414 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495416 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495419 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495423 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495427 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495430 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495433 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495437 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495440 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495443 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.495445 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 10:01:30.500501 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495520 2576 flags.go:64] FLAG: --address="0.0.0.0" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495552 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495558 2576 flags.go:64] FLAG: --anonymous-auth="true" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495562 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495583 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495587 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495591 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495595 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495599 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495602 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495605 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495609 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495612 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495628 2576 flags.go:64] FLAG: --cgroup-root="" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495631 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495634 2576 flags.go:64] FLAG: --client-ca-file="" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495637 2576 flags.go:64] FLAG: --cloud-provider="external" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495640 2576 flags.go:64] FLAG: --cluster-dns="[]" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495645 2576 flags.go:64] FLAG: --cluster-domain="" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495648 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495651 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495654 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495658 2576 flags.go:64] FLAG: --container-log-max-files="5" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495661 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 08 10:01:30.500995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495664 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495667 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495671 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495674 2576 flags.go:64] FLAG: --contention-profiling="false" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495678 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495681 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495684 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495687 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495691 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495694 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495697 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495699 2576 flags.go:64] FLAG: --enable-load-reader="false" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495702 2576 flags.go:64] FLAG: --enable-server="true" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495712 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495721 2576 flags.go:64] FLAG: --event-burst="100" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495724 2576 flags.go:64] FLAG: --event-qps="50" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495727 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495730 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495733 2576 flags.go:64] FLAG: --eviction-hard="" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495736 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495739 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495742 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495745 2576 flags.go:64] FLAG: --eviction-soft="" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495748 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495750 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 08 10:01:30.501667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495753 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495756 2576 flags.go:64] FLAG: --experimental-mounter-path="" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495759 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495761 2576 flags.go:64] FLAG: --fail-swap-on="true" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495764 2576 flags.go:64] FLAG: --feature-gates="" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495768 2576 flags.go:64] FLAG: --file-check-frequency="20s" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495771 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495774 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495777 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495780 2576 flags.go:64] FLAG: --healthz-port="10248" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495783 2576 flags.go:64] FLAG: --help="false" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495787 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-132-206.ec2.internal" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495790 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495793 2576 flags.go:64] FLAG: --http-check-frequency="20s" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495795 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495799 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495802 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495805 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495808 2576 flags.go:64] FLAG: --image-service-endpoint="" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495811 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495813 2576 flags.go:64] FLAG: --kube-api-burst="100" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495818 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495821 2576 flags.go:64] FLAG: --kube-api-qps="50" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495824 2576 flags.go:64] FLAG: --kube-reserved="" Sep 08 10:01:30.502295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495827 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495830 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495833 2576 flags.go:64] FLAG: --kubelet-cgroups="" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495836 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495839 2576 flags.go:64] FLAG: --lock-file="" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495842 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495845 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495847 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495853 2576 flags.go:64] FLAG: --log-json-split-stream="false" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495855 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495858 2576 flags.go:64] FLAG: --log-text-split-stream="false" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495861 2576 flags.go:64] FLAG: --logging-format="text" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495864 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495868 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495870 2576 flags.go:64] FLAG: --manifest-url="" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495873 2576 flags.go:64] FLAG: --manifest-url-header="" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495878 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495883 2576 flags.go:64] FLAG: --max-open-files="1000000" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495887 2576 flags.go:64] FLAG: --max-pods="110" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495892 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495895 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495898 2576 flags.go:64] FLAG: --memory-manager-policy="None" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495901 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495904 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495907 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 08 10:01:30.503125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495910 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495916 2576 flags.go:64] FLAG: --node-status-max-images="50" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495919 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495922 2576 flags.go:64] FLAG: --oom-score-adj="-999" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495926 2576 flags.go:64] FLAG: --pod-cidr="" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495935 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495942 2576 flags.go:64] FLAG: --pod-manifest-path="" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495945 2576 flags.go:64] FLAG: --pod-max-pids="-1" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495948 2576 flags.go:64] FLAG: --pods-per-core="0" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495952 2576 flags.go:64] FLAG: --port="10250" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495955 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495957 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-057c24b72061b0f4d" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495960 2576 flags.go:64] FLAG: --qos-reserved="" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495963 2576 flags.go:64] FLAG: --read-only-port="10255" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495966 2576 flags.go:64] FLAG: --register-node="true" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495969 2576 flags.go:64] FLAG: --register-with-taints="" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495973 2576 flags.go:64] FLAG: --registry-burst="10" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495976 2576 flags.go:64] FLAG: --registry-qps="5" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495978 2576 flags.go:64] FLAG: --reserved-cpus="" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495981 2576 flags.go:64] FLAG: --reserved-memory="" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495985 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495988 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495991 2576 flags.go:64] FLAG: --rotate-certificates="false" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495994 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495996 2576 flags.go:64] FLAG: --runonce="false" Sep 08 10:01:30.504248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.495999 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496004 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496007 2576 flags.go:64] FLAG: --seccomp-default="false" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496009 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496012 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496015 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496018 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496021 2576 flags.go:64] FLAG: --storage-driver-password="root" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496024 2576 flags.go:64] FLAG: --storage-driver-secure="false" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496026 2576 flags.go:64] FLAG: --storage-driver-table="stats" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496029 2576 flags.go:64] FLAG: --storage-driver-user="root" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496032 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496037 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496041 2576 flags.go:64] FLAG: --system-cgroups="" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496043 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496048 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496051 2576 flags.go:64] FLAG: --tls-cert-file="" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496054 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496061 2576 flags.go:64] FLAG: --tls-min-version="" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496064 2576 flags.go:64] FLAG: --tls-private-key-file="" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496066 2576 flags.go:64] FLAG: --topology-manager-policy="none" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496069 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496072 2576 flags.go:64] FLAG: --topology-manager-scope="container" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496075 2576 flags.go:64] FLAG: --v="2" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496079 2576 flags.go:64] FLAG: --version="false" Sep 08 10:01:30.505337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496083 2576 flags.go:64] FLAG: --vmodule="" Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496086 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.496089 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496218 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496221 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496224 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496227 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496230 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496234 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496237 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496240 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496242 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496245 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496247 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496250 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496252 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496255 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496257 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496260 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496264 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496267 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 10:01:30.506010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496277 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496280 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496282 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496285 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496287 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496290 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496293 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496295 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496298 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496300 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496303 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496305 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496308 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496311 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496314 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496316 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496319 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496321 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496324 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496327 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 10:01:30.506550 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496329 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496333 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496337 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496340 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496343 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496346 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496349 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496352 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496355 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496357 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496363 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496367 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496371 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496382 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496386 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496389 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496392 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496395 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496397 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 10:01:30.507244 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496400 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496403 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496406 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496410 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496413 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496417 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496420 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496422 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496425 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496428 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496431 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496434 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496436 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496439 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496441 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496444 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496447 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496449 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496452 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 10:01:30.508113 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496454 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496457 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496460 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496462 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496467 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496469 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496472 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496474 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496483 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496486 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496488 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496491 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496494 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496496 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496499 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496502 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496504 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496507 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496510 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496518 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 10:01:30.508778 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496521 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 10:01:30.509345 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496523 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 10:01:30.509345 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.496526 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 10:01:30.509345 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.497262 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 10:01:30.509345 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.505338 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 08 10:01:30.509345 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.505354 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 08 10:01:30.509345 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.506871 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 08 10:01:30.509345 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.506886 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 08 10:01:30.509345 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507001 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 10:01:30.509345 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507010 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 10:01:30.509345 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507016 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 10:01:30.509345 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507021 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 10:01:30.509345 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507026 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 10:01:30.509345 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507031 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 10:01:30.509345 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507036 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 10:01:30.509345 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507040 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507044 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507049 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507054 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507058 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507064 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507068 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507073 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507077 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507082 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507086 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507090 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507094 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507098 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507102 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507107 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507111 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507115 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507119 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507123 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 10:01:30.509754 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507127 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507131 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507136 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507141 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507145 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507149 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507154 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507158 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507162 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507166 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507170 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507174 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507178 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507182 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507186 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507190 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507194 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507201 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507209 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507214 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 10:01:30.510540 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507219 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507224 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507229 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507240 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507245 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507249 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507253 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507257 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507261 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507265 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507269 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507273 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507277 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507282 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507287 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507291 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507296 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507300 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507304 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507309 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 10:01:30.511339 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507315 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507320 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507325 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507331 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507336 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507341 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507346 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507350 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507355 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507360 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507364 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507369 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507373 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507377 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507381 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507386 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507391 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507395 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507401 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 10:01:30.512137 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507406 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 10:01:30.512609 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507410 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 10:01:30.512609 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507415 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 10:01:30.512609 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507419 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 10:01:30.512609 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507423 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 08 10:01:30.512609 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507427 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 10:01:30.512609 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507431 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 10:01:30.512609 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507436 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 10:01:30.512609 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507440 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 10:01:30.512609 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507444 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 10:01:30.512609 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507447 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 10:01:30.512609 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507451 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 10:01:30.512609 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507456 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 10:01:30.512609 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.507464 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 10:01:30.512609 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507702 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 10:01:30.512609 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507712 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507717 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507721 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507725 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507730 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507734 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507738 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507743 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507747 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507751 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507756 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507761 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507765 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507769 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507773 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507777 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507782 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507787 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507791 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507795 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 10:01:30.513159 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507799 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507804 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507810 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507816 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507821 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507826 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507831 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507835 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507840 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507844 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507848 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507852 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507856 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507860 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507864 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507868 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507873 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507877 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507881 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507885 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 10:01:30.513745 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507889 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507893 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507898 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507903 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507910 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507915 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507920 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507925 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507929 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507933 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507937 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507941 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507946 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507951 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507955 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507959 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507963 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507967 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507973 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507978 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 10:01:30.514420 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507982 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507987 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507990 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507994 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.507999 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508003 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508007 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508011 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508015 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508019 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508024 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508027 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508031 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508036 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508040 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508044 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508049 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508053 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508057 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508061 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 10:01:30.514941 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508066 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508069 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508076 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508081 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508085 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508089 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508093 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508098 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508103 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508107 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508111 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508115 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508119 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508122 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508126 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508130 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508134 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 10:01:30.515545 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:30.508138 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 10:01:30.516033 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.508145 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 10:01:30.516033 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.508299 2576 server.go:962] "Client rotation is on, will bootstrap in background" Sep 08 10:01:30.516033 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.513644 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 08 10:01:30.516033 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.514872 2576 server.go:1019] "Starting client certificate rotation" Sep 08 10:01:30.516033 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.514987 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 10:01:30.516033 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.515072 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 10:01:30.517240 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.517223 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 10:01:30.518400 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.518382 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 10:01:30.526916 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.526899 2576 log.go:25] "Validated CRI v1 runtime API" Sep 08 10:01:30.533700 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.533685 2576 log.go:25] "Validated CRI v1 image API" Sep 08 10:01:30.534546 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.534534 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 08 10:01:30.537595 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.537573 2576 fs.go:135] Filesystem UUIDs: map[2bceb35f-1472-45dd-a831-af68a9fd8d5c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7a9b7cac-1fcb-4857-98e4-434d58349c34:/dev/nvme0n1p4] Sep 08 10:01:30.537682 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.537594 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 08 10:01:30.542589 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.542479 2576 manager.go:216] Machine: {Timestamp:2026-09-08 10:01:30.541203335 +0000 UTC m=+0.330255130 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096432 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23fb325003d479981b13910dc7e4f0 SystemUUID:ec23fb32-5003-d479-981b-13910dc7e4f0 BootID:d4b62066-4ba7-4c4a-85eb-0ec4dc1d0430 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:df:4f:7f:59:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:df:4f:7f:59:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:6b:db:32:c1:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 08 10:01:30.542589 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.542586 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 08 10:01:30.542716 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.542684 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 08 10:01:30.545399 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.545373 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 08 10:01:30.545554 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.545405 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-206.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 08 10:01:30.546177 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.546167 2576 topology_manager.go:138] "Creating topology manager with none policy" Sep 08 10:01:30.546210 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.546181 2576 container_manager_linux.go:309] "Creating device plugin manager" Sep 08 10:01:30.546210 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.546189 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 10:01:30.546818 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.546808 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 10:01:30.547537 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.547526 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 08 10:01:30.550200 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.550184 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 10:01:30.552684 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.552672 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 08 10:01:30.552810 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.552801 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 08 10:01:30.552844 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.552834 2576 kubelet.go:486] "Attempting to sync node with API server" Sep 08 10:01:30.552875 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.552844 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 08 10:01:30.552875 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.552858 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 08 10:01:30.552875 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.552867 2576 kubelet.go:398] "Adding apiserver pod source" Sep 08 10:01:30.552875 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.552876 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 08 10:01:30.553369 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.553346 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 10:01:30.553471 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.553383 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 10:01:30.555474 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.555459 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 08 10:01:30.559114 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.559087 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 08 10:01:30.559195 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.559139 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 08 10:01:30.560400 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.560378 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 08 10:01:30.560488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.560406 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 08 10:01:30.560488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.560416 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 08 10:01:30.560488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.560426 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 08 10:01:30.560488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.560445 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 08 10:01:30.560488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.560454 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 08 10:01:30.560488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.560474 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 08 10:01:30.560488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.560488 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 08 10:01:30.560707 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.560499 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 08 10:01:30.560707 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.560507 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 08 10:01:30.560707 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.560516 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 08 10:01:30.560707 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.560522 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 08 10:01:30.561294 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.561277 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 08 10:01:30.561294 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.561294 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 08 10:01:30.563758 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.563734 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 08 10:01:30.563758 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.563745 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-206.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 08 10:01:30.563855 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.563738 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-206.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 08 10:01:30.565462 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.565449 2576 server.go:1268] "Started kubelet" Sep 08 10:01:30.565577 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.565539 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 08 10:01:30.565709 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.565529 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 08 10:01:30.565809 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.565731 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 08 10:01:30.566408 ip-10-0-132-206 systemd[1]: Started Kubernetes Kubelet. Sep 08 10:01:30.567434 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.567362 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 08 10:01:30.567657 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.567641 2576 server.go:310] "Adding debug handlers to kubelet server" Sep 08 10:01:30.572182 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.572161 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 08 10:01:30.572244 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.572180 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 08 10:01:30.572375 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.572287 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-206.ec2.internal\" not found" Sep 08 10:01:30.573016 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.571495 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-206.ec2.internal.18d3503ad274e674 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-206.ec2.internal,UID:ip-10-0-132-206.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-206.ec2.internal,},FirstTimestamp:2026-09-08 10:01:30.56542066 +0000 UTC m=+0.354472454,LastTimestamp:2026-09-08 10:01:30.56542066 +0000 UTC m=+0.354472454,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-206.ec2.internal,}" Sep 08 10:01:30.573016 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.572971 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 08 10:01:30.573016 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.573011 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 08 10:01:30.573235 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.573217 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 08 10:01:30.573289 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.573239 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 08 10:01:30.573344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.573329 2576 factory.go:55] Registering systemd factory Sep 08 10:01:30.573395 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.573345 2576 factory.go:223] Registration of the systemd container factory successfully Sep 08 10:01:30.573437 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.573422 2576 reconstruct.go:97] "Volume reconstruction finished" Sep 08 10:01:30.573437 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.573432 2576 reconciler.go:29] "Reconciler: start to sync state" Sep 08 10:01:30.573564 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.573552 2576 factory.go:153] Registering CRI-O factory Sep 08 10:01:30.573635 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.573569 2576 factory.go:223] Registration of the crio container factory successfully Sep 08 10:01:30.573680 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.573636 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 08 10:01:30.573680 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.573655 2576 factory.go:103] Registering Raw factory Sep 08 10:01:30.573680 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.573673 2576 manager.go:1196] Started watching for new ooms in manager Sep 08 10:01:30.574051 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.574040 2576 manager.go:349] Starting recovery of all containers Sep 08 10:01:30.576100 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.576078 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xclgv" Sep 08 10:01:30.578295 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.578273 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 08 10:01:30.578295 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.578272 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-206.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 08 10:01:30.580087 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.580061 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 08 10:01:30.582952 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.582924 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xclgv" Sep 08 10:01:30.584098 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.584075 2576 manager.go:354] Recovery completed Sep 08 10:01:30.590593 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.590468 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 10:01:30.592607 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.592586 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasSufficientMemory" Sep 08 10:01:30.592687 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.592632 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 10:01:30.592687 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.592648 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasSufficientPID" Sep 08 10:01:30.593240 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.593224 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 08 10:01:30.593240 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.593233 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 10:01:30.593240 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.593238 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 08 10:01:30.593404 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.593255 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 08 10:01:30.593954 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.593941 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasSufficientMemory" Sep 08 10:01:30.594019 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.593967 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 10:01:30.594019 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.593977 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasSufficientPID" Sep 08 10:01:30.595647 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.594535 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-206.ec2.internal.18d3503ad413afda default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-206.ec2.internal,UID:ip-10-0-132-206.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-206.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-206.ec2.internal,},FirstTimestamp:2026-09-08 10:01:30.592604122 +0000 UTC m=+0.381655919,LastTimestamp:2026-09-08 10:01:30.592604122 +0000 UTC m=+0.381655919,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-206.ec2.internal,}" Sep 08 10:01:30.595812 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.595243 2576 policy_none.go:49] "None policy: Start" Sep 08 10:01:30.595812 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.595798 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 08 10:01:30.595900 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.595816 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 08 10:01:30.597792 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.597767 2576 policy_none.go:47] "Start" Sep 08 10:01:30.634330 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.634313 2576 manager.go:337] "Starting Device Plugin manager" Sep 08 10:01:30.651194 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.634366 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 08 10:01:30.651194 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.634378 2576 server.go:87] "Starting device plugin registration server" Sep 08 10:01:30.651194 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.634771 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 08 10:01:30.651194 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.634783 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 08 10:01:30.651194 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.634884 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 08 10:01:30.651194 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.634963 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 08 10:01:30.651194 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.634971 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 08 10:01:30.651194 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.635770 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 08 10:01:30.651194 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.635819 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-206.ec2.internal\" not found" Sep 08 10:01:30.670873 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.670845 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 08 10:01:30.670972 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.670895 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 08 10:01:30.670972 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.670916 2576 kubelet.go:2448] "Starting kubelet main sync loop" Sep 08 10:01:30.670972 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.670954 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 08 10:01:30.673458 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.673441 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 10:01:30.735548 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.735502 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 10:01:30.736605 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.736585 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasSufficientMemory" Sep 08 10:01:30.736686 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.736634 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 10:01:30.736686 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.736651 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasSufficientPID" Sep 08 10:01:30.736686 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.736674 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-206.ec2.internal" Sep 08 10:01:30.744723 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.744654 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-206.ec2.internal" Sep 08 10:01:30.744723 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.744678 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-206.ec2.internal\": node \"ip-10-0-132-206.ec2.internal\" not found" Sep 08 10:01:30.761166 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.761147 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-206.ec2.internal\" not found" Sep 08 10:01:30.771377 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.771352 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-206.ec2.internal"] Sep 08 10:01:30.771454 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.771415 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 10:01:30.772835 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.772819 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasSufficientMemory" Sep 08 10:01:30.772913 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.772850 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 10:01:30.772913 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.772861 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasSufficientPID" Sep 08 10:01:30.773956 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.773940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/19e5ada69ca2575a19d6db1fc5d72bd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal\" (UID: \"19e5ada69ca2575a19d6db1fc5d72bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal" Sep 08 10:01:30.774040 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.773962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19e5ada69ca2575a19d6db1fc5d72bd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal\" (UID: \"19e5ada69ca2575a19d6db1fc5d72bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal" Sep 08 10:01:30.775025 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.775011 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 10:01:30.775142 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.775127 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal" Sep 08 10:01:30.775197 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.775165 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 10:01:30.775785 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.775770 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasSufficientMemory" Sep 08 10:01:30.775785 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.775782 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasSufficientMemory" Sep 08 10:01:30.775915 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.775795 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 10:01:30.775915 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.775806 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 10:01:30.775915 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.775810 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasSufficientPID" Sep 08 10:01:30.775915 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.775821 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasSufficientPID" Sep 08 10:01:30.777989 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.777975 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-206.ec2.internal" Sep 08 10:01:30.778052 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.777999 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 10:01:30.778720 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.778706 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasSufficientMemory" Sep 08 10:01:30.778782 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.778731 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 10:01:30.778782 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.778743 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeHasSufficientPID" Sep 08 10:01:30.792168 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.792152 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-206.ec2.internal\" not found" node="ip-10-0-132-206.ec2.internal" Sep 08 10:01:30.795559 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.795543 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-206.ec2.internal\" not found" node="ip-10-0-132-206.ec2.internal" Sep 08 10:01:30.862008 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.861975 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-206.ec2.internal\" not found" Sep 08 10:01:30.874110 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.874083 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aee6609005f8620b26930034856bde0d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-206.ec2.internal\" (UID: \"aee6609005f8620b26930034856bde0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-206.ec2.internal" Sep 08 10:01:30.874232 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.874132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/19e5ada69ca2575a19d6db1fc5d72bd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal\" (UID: \"19e5ada69ca2575a19d6db1fc5d72bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal" Sep 08 10:01:30.874232 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.874157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19e5ada69ca2575a19d6db1fc5d72bd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal\" (UID: \"19e5ada69ca2575a19d6db1fc5d72bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal" Sep 08 10:01:30.874232 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.874192 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/19e5ada69ca2575a19d6db1fc5d72bd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal\" (UID: \"19e5ada69ca2575a19d6db1fc5d72bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal" Sep 08 10:01:30.874232 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.874192 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19e5ada69ca2575a19d6db1fc5d72bd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal\" (UID: \"19e5ada69ca2575a19d6db1fc5d72bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal" Sep 08 10:01:30.962325 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:30.962280 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-206.ec2.internal\" not found" Sep 08 10:01:30.974575 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.974558 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aee6609005f8620b26930034856bde0d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-206.ec2.internal\" (UID: \"aee6609005f8620b26930034856bde0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-206.ec2.internal" Sep 08 10:01:30.974633 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:30.974592 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aee6609005f8620b26930034856bde0d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-206.ec2.internal\" (UID: \"aee6609005f8620b26930034856bde0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-206.ec2.internal" Sep 08 10:01:31.063036 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:31.062976 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-206.ec2.internal\" not found" Sep 08 10:01:31.094463 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.094438 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal" Sep 08 10:01:31.099291 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.099227 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-206.ec2.internal" Sep 08 10:01:31.163990 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:31.163956 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-206.ec2.internal\" not found" Sep 08 10:01:31.264480 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:31.264435 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-206.ec2.internal\" not found" Sep 08 10:01:31.365052 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:31.364991 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-206.ec2.internal\" not found" Sep 08 10:01:31.431437 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.431413 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 10:01:31.465225 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:31.465201 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-206.ec2.internal\" not found" Sep 08 10:01:31.514594 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.514563 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 08 10:01:31.515104 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.514733 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 10:01:31.515104 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.514738 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 10:01:31.566479 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:31.566452 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-206.ec2.internal\" not found" Sep 08 10:01:31.588209 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.588161 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 09:56:30 +0000 UTC" deadline="2028-05-29 12:08:10.803811156 +0000 UTC" Sep 08 10:01:31.588209 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.588196 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15098h6m39.215619357s" Sep 08 10:01:31.639824 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.639763 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 10:01:31.651076 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:31.651047 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee6609005f8620b26930034856bde0d.slice/crio-ff59648c51e9718b8c4226d46845d1830396ff66b79ad16df2207158914a06ef WatchSource:0}: Error finding container ff59648c51e9718b8c4226d46845d1830396ff66b79ad16df2207158914a06ef: Status 404 returned error can't find the container with id ff59648c51e9718b8c4226d46845d1830396ff66b79ad16df2207158914a06ef Sep 08 10:01:31.660567 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.660554 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 10:01:31.667316 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:31.667298 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e5ada69ca2575a19d6db1fc5d72bd7.slice/crio-a2450bd991ae446cbbc42efba28c09a17bb9c767c57d300bfffce8df206108f6 WatchSource:0}: Error finding container a2450bd991ae446cbbc42efba28c09a17bb9c767c57d300bfffce8df206108f6: Status 404 returned error can't find the container with id a2450bd991ae446cbbc42efba28c09a17bb9c767c57d300bfffce8df206108f6 Sep 08 10:01:31.672385 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.672344 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-206.ec2.internal" Sep 08 10:01:31.674397 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.674359 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal" event={"ID":"19e5ada69ca2575a19d6db1fc5d72bd7","Type":"ContainerStarted","Data":"a2450bd991ae446cbbc42efba28c09a17bb9c767c57d300bfffce8df206108f6"} Sep 08 10:01:31.675268 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.675252 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-206.ec2.internal" event={"ID":"aee6609005f8620b26930034856bde0d","Type":"ContainerStarted","Data":"ff59648c51e9718b8c4226d46845d1830396ff66b79ad16df2207158914a06ef"} Sep 08 10:01:31.677914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.677902 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 10:01:31.679562 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.679550 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal" Sep 08 10:01:31.688347 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.688333 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 10:01:31.877348 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:31.877322 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 10:01:32.554230 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.554199 2576 apiserver.go:52] "Watching apiserver" Sep 08 10:01:32.561785 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.561760 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 08 10:01:32.562127 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.562093 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-ktdpl","kube-system/konnectivity-agent-9g9nb","kube-system/kube-apiserver-proxy-ip-10-0-132-206.ec2.internal","openshift-multus/multus-additional-cni-plugins-5mv87","openshift-multus/network-metrics-daemon-jgstq","openshift-network-diagnostics/network-check-target-mvf7l","openshift-ovn-kubernetes/ovnkube-node-4fcgv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4","openshift-cluster-node-tuning-operator/tuned-lsd8z","openshift-dns/node-resolver-95fpk","openshift-image-registry/node-ca-wqx5v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal","openshift-multus/multus-28d92","openshift-network-operator/iptables-alerter-qhsv4"] Sep 08 10:01:32.564475 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.564454 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.566760 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.566561 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-28d92" Sep 08 10:01:32.566760 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.566714 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 08 10:01:32.566887 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.566757 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 08 10:01:32.566887 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.566721 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vzkdv\"" Sep 08 10:01:32.567452 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.567428 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 08 10:01:32.567535 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.567431 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 08 10:01:32.567535 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.567431 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 08 10:01:32.567535 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.567432 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 08 10:01:32.568489 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.568460 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c2zdm\"" Sep 08 10:01:32.568698 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.568682 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qhsv4" Sep 08 10:01:32.569099 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.569085 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 08 10:01:32.569188 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.569099 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 08 10:01:32.569188 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.569118 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 08 10:01:32.569304 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.569196 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 08 10:01:32.570936 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.570785 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 08 10:01:32.570936 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.570785 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 08 10:01:32.570936 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.570838 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qs7sj\"" Sep 08 10:01:32.571115 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.570943 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.571115 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.571057 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 08 10:01:32.572577 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.572562 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 08 10:01:32.572836 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.572818 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ng9q2\"" Sep 08 10:01:32.572908 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.572826 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 08 10:01:32.572908 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.572859 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 08 10:01:32.573233 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.573215 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:32.573315 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:32.573289 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:01:32.575532 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.575513 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:32.575635 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:32.575568 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:01:32.579132 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.579111 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.581315 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.581008 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 08 10:01:32.581315 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.581116 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2dftb\"" Sep 08 10:01:32.581315 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.581207 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:32.581315 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.581249 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 08 10:01:32.581315 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:32.581259 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:01:32.582253 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582227 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a73b204-193b-48be-9f93-a6cf774ec172-ovnkube-config\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.582341 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582257 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-system-cni-dir\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.582341 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce8608f0-a21a-45fb-affd-0857e3802aca-cni-binary-copy\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.582341 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-node-log\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.582341 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-kubelet\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.582559 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582344 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-run-systemd\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.582559 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582365 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-etc-openvswitch\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.582559 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582404 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-run-openvswitch\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.582559 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a73b204-193b-48be-9f93-a6cf774ec172-ovnkube-script-lib\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.582559 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-run-ovn-kubernetes\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.582559 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582476 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-cni-bin\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.582559 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grfx9\" (UniqueName: \"kubernetes.io/projected/ce8608f0-a21a-45fb-affd-0857e3802aca-kube-api-access-grfx9\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.582559 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa2fe904-ae11-4f90-b530-f9bbcf80937d-iptables-alerter-script\") pod \"iptables-alerter-qhsv4\" (UID: \"aa2fe904-ae11-4f90-b530-f9bbcf80937d\") " pod="openshift-network-operator/iptables-alerter-qhsv4" Sep 08 10:01:32.582559 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582532 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa2fe904-ae11-4f90-b530-f9bbcf80937d-host-slash\") pod \"iptables-alerter-qhsv4\" (UID: \"aa2fe904-ae11-4f90-b530-f9bbcf80937d\") " pod="openshift-network-operator/iptables-alerter-qhsv4" Sep 08 10:01:32.582559 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582550 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b9c86ed-68b3-4864-9824-8e55025d48ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-run-ovn\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582592 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-slash\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a73b204-193b-48be-9f93-a6cf774ec172-env-overrides\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m66r8\" (UniqueName: \"kubernetes.io/projected/1a73b204-193b-48be-9f93-a6cf774ec172-kube-api-access-m66r8\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-run-netns\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-var-lib-cni-multus\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582747 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b9c86ed-68b3-4864-9824-8e55025d48ba-cnibin\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b9c86ed-68b3-4864-9824-8e55025d48ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-multus-cni-dir\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-multus-conf-dir\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582862 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-etc-kubernetes\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582892 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b9c86ed-68b3-4864-9824-8e55025d48ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn2gw\" (UniqueName: \"kubernetes.io/projected/01480054-d16a-4820-af73-7a493b53ead0-kube-api-access-xn2gw\") pod \"network-metrics-daemon-jgstq\" (UID: \"01480054-d16a-4820-af73-7a493b53ead0\") " pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-cnibin\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.582986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-multus-socket-dir-parent\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.583034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583008 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-var-lib-kubelet\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce8608f0-a21a-45fb-affd-0857e3802aca-multus-daemon-config\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs\") pod \"network-metrics-daemon-jgstq\" (UID: \"01480054-d16a-4820-af73-7a493b53ead0\") " pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583093 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-cni-netd\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a73b204-193b-48be-9f93-a6cf774ec172-ovn-node-metrics-cert\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-hostroot\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czmhr\" (UniqueName: \"kubernetes.io/projected/3b9c86ed-68b3-4864-9824-8e55025d48ba-kube-api-access-czmhr\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583181 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-log-socket\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583185 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9g9nb" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583203 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-systemd-units\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-run-netns\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583242 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-var-lib-cni-bin\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583262 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nbww\" (UniqueName: \"kubernetes.io/projected/aa2fe904-ae11-4f90-b530-f9bbcf80937d-kube-api-access-2nbww\") pod \"iptables-alerter-qhsv4\" (UID: \"aa2fe904-ae11-4f90-b530-f9bbcf80937d\") " pod="openshift-network-operator/iptables-alerter-qhsv4" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583282 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-var-lib-openvswitch\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-run-k8s-cni-cncf-io\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-run-multus-certs\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.583793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583351 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b9c86ed-68b3-4864-9824-8e55025d48ba-system-cni-dir\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.584353 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.583375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b9c86ed-68b3-4864-9824-8e55025d48ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.585083 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.585067 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 08 10:01:32.585157 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.585069 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x28rs\"" Sep 08 10:01:32.585157 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.585074 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 08 10:01:32.585447 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.585432 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-95fpk" Sep 08 10:01:32.587255 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.587240 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 08 10:01:32.587349 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.587320 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 08 10:01:32.587349 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.587330 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fxg4d\"" Sep 08 10:01:32.587642 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.587625 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wqx5v" Sep 08 10:01:32.589272 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.589254 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 08 10:01:32.589772 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.589713 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5x2pn\"" Sep 08 10:01:32.589772 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.589758 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 08 10:01:32.589907 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.589747 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 08 10:01:32.589907 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.589857 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.591779 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.591757 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 08 10:01:32.591864 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.591784 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 08 10:01:32.591864 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.591801 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 08 10:01:32.591864 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.591801 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n9jht\"" Sep 08 10:01:32.598935 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.598916 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-626gq" Sep 08 10:01:32.603733 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.603715 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-626gq" Sep 08 10:01:32.674904 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.674879 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 08 10:01:32.684138 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-cni-bin\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.684276 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce8608f0-a21a-45fb-affd-0857e3802aca-cni-binary-copy\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.684276 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-run-systemd\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.684276 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dkq8\" (UniqueName: \"kubernetes.io/projected/02dac30c-8810-4609-8768-4047fd4405da-kube-api-access-5dkq8\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.684276 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa2fe904-ae11-4f90-b530-f9bbcf80937d-host-slash\") pod \"iptables-alerter-qhsv4\" (UID: \"aa2fe904-ae11-4f90-b530-f9bbcf80937d\") " pod="openshift-network-operator/iptables-alerter-qhsv4" Sep 08 10:01:32.684276 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684266 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b9c86ed-68b3-4864-9824-8e55025d48ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.684540 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs\") pod \"network-metrics-daemon-jgstq\" (UID: \"01480054-d16a-4820-af73-7a493b53ead0\") " pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:32.684540 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684289 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-run-systemd\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.684540 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684278 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-cni-bin\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.684540 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684331 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa2fe904-ae11-4f90-b530-f9bbcf80937d-host-slash\") pod \"iptables-alerter-qhsv4\" (UID: \"aa2fe904-ae11-4f90-b530-f9bbcf80937d\") " pod="openshift-network-operator/iptables-alerter-qhsv4" Sep 08 10:01:32.684765 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:32.684547 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 10:01:32.684765 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:32.684661 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs podName:01480054-d16a-4820-af73-7a493b53ead0 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:33.184641093 +0000 UTC m=+2.973692892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs") pod "network-metrics-daemon-jgstq" (UID: "01480054-d16a-4820-af73-7a493b53ead0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 10:01:32.684765 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-run\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.684765 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12807ade-0ca9-4bd3-b277-afba07e51a26-host\") pod \"node-ca-wqx5v\" (UID: \"12807ade-0ca9-4bd3-b277-afba07e51a26\") " pod="openshift-image-registry/node-ca-wqx5v" Sep 08 10:01:32.684765 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-run-ovn\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.684765 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a73b204-193b-48be-9f93-a6cf774ec172-env-overrides\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.684965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684768 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-run-netns\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.684965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce8608f0-a21a-45fb-affd-0857e3802aca-cni-binary-copy\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.684965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b9c86ed-68b3-4864-9824-8e55025d48ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.684965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684816 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-run-ovn\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.684965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-var-lib-kubelet\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.684965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-host\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.684965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-multus-cni-dir\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.684965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-etc-kubernetes\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.684965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684930 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-kubernetes\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.684965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684941 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b9c86ed-68b3-4864-9824-8e55025d48ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.685252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684944 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d180476a-ce29-4508-80fe-3151edbba85e-etc-tuned\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.685252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.684985 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d180476a-ce29-4508-80fe-3151edbba85e-tmp\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.685252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.685252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685013 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-run-netns\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.685252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce8608f0-a21a-45fb-affd-0857e3802aca-multus-daemon-config\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.685252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d5114137-2635-4a57-bdae-7866a1b417a2-hosts-file\") pod \"node-resolver-95fpk\" (UID: \"d5114137-2635-4a57-bdae-7866a1b417a2\") " pod="openshift-dns/node-resolver-95fpk" Sep 08 10:01:32.685252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685052 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-etc-kubernetes\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.685252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685089 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-multus-cni-dir\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.685252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685106 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b95h\" (UniqueName: \"kubernetes.io/projected/d5114137-2635-4a57-bdae-7866a1b417a2-kube-api-access-7b95h\") pod \"node-resolver-95fpk\" (UID: \"d5114137-2635-4a57-bdae-7866a1b417a2\") " pod="openshift-dns/node-resolver-95fpk" Sep 08 10:01:32.685252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685115 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.685252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9nw\" (UniqueName: \"kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw\") pod \"network-check-target-mvf7l\" (UID: \"334662a7-e44f-4aa9-8304-0a13dfed33e7\") " pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:32.685252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685163 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b9c86ed-68b3-4864-9824-8e55025d48ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.685252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685204 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a73b204-193b-48be-9f93-a6cf774ec172-env-overrides\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.685252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-socket-dir\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.685252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-cni-netd\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-hostroot\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685305 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-cni-netd\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czmhr\" (UniqueName: \"kubernetes.io/projected/3b9c86ed-68b3-4864-9824-8e55025d48ba-kube-api-access-czmhr\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685410 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-hostroot\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685438 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-sys\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685459 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce8608f0-a21a-45fb-affd-0857e3802aca-multus-daemon-config\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b7ee997-0723-4a47-90b5-805cd7721382-kubelet-config\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-log-socket\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-systemd-units\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685553 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-log-socket\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-run-netns\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nbww\" (UniqueName: \"kubernetes.io/projected/aa2fe904-ae11-4f90-b530-f9bbcf80937d-kube-api-access-2nbww\") pod \"iptables-alerter-qhsv4\" (UID: \"aa2fe904-ae11-4f90-b530-f9bbcf80937d\") " pod="openshift-network-operator/iptables-alerter-qhsv4" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685643 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-systemd-units\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-modprobe-d\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685678 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-run-netns\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685783 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-sysctl-d\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.685862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9dpn\" (UniqueName: \"kubernetes.io/projected/12807ade-0ca9-4bd3-b277-afba07e51a26-kube-api-access-c9dpn\") pod \"node-ca-wqx5v\" (UID: \"12807ade-0ca9-4bd3-b277-afba07e51a26\") " pod="openshift-image-registry/node-ca-wqx5v" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-var-lib-openvswitch\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-run-k8s-cni-cncf-io\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685905 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b9c86ed-68b3-4864-9824-8e55025d48ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-sysctl-conf\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685948 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5pwp\" (UniqueName: \"kubernetes.io/projected/d180476a-ce29-4508-80fe-3151edbba85e-kube-api-access-r5pwp\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685958 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-run-k8s-cni-cncf-io\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-run-ovn-kubernetes\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.685937 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-var-lib-openvswitch\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa2fe904-ae11-4f90-b530-f9bbcf80937d-iptables-alerter-script\") pod \"iptables-alerter-qhsv4\" (UID: \"aa2fe904-ae11-4f90-b530-f9bbcf80937d\") " pod="openshift-network-operator/iptables-alerter-qhsv4" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686017 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-run-ovn-kubernetes\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn2gw\" (UniqueName: \"kubernetes.io/projected/01480054-d16a-4820-af73-7a493b53ead0-kube-api-access-xn2gw\") pod \"network-metrics-daemon-jgstq\" (UID: \"01480054-d16a-4820-af73-7a493b53ead0\") " pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a73b204-193b-48be-9f93-a6cf774ec172-ovnkube-config\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-system-cni-dir\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-node-log\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-kubelet\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-etc-openvswitch\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.686496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-run-openvswitch\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a73b204-193b-48be-9f93-a6cf774ec172-ovnkube-script-lib\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grfx9\" (UniqueName: \"kubernetes.io/projected/ce8608f0-a21a-45fb-affd-0857e3802aca-kube-api-access-grfx9\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686243 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-systemd\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686244 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-node-log\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686288 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12807ade-0ca9-4bd3-b277-afba07e51a26-serviceca\") pod \"node-ca-wqx5v\" (UID: \"12807ade-0ca9-4bd3-b277-afba07e51a26\") " pod="openshift-image-registry/node-ca-wqx5v" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-device-dir\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-etc-selinux\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686359 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-slash\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m66r8\" (UniqueName: \"kubernetes.io/projected/1a73b204-193b-48be-9f93-a6cf774ec172-kube-api-access-m66r8\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-var-lib-cni-multus\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686426 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b9c86ed-68b3-4864-9824-8e55025d48ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b9c86ed-68b3-4864-9824-8e55025d48ba-cnibin\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-multus-conf-dir\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b9c86ed-68b3-4864-9824-8e55025d48ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-sysconfig\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.687344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686538 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa2fe904-ae11-4f90-b530-f9bbcf80937d-iptables-alerter-script\") pod \"iptables-alerter-qhsv4\" (UID: \"aa2fe904-ae11-4f90-b530-f9bbcf80937d\") " pod="openshift-network-operator/iptables-alerter-qhsv4" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686551 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-system-cni-dir\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686540 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ed41459-ca2d-4277-a2df-c7879f425183-konnectivity-ca\") pod \"konnectivity-agent-9g9nb\" (UID: \"1ed41459-ca2d-4277-a2df-c7879f425183\") " pod="kube-system/konnectivity-agent-9g9nb" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-cnibin\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686605 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-kubelet\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686602 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-etc-openvswitch\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-multus-socket-dir-parent\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686666 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-run-openvswitch\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686683 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-multus-socket-dir-parent\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686701 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a73b204-193b-48be-9f93-a6cf774ec172-ovnkube-config\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686706 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a73b204-193b-48be-9f93-a6cf774ec172-host-slash\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686732 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-cnibin\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686879 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-var-lib-cni-multus\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686892 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b9c86ed-68b3-4864-9824-8e55025d48ba-cnibin\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686940 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-multus-conf-dir\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-var-lib-kubelet\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.686999 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ed41459-ca2d-4277-a2df-c7879f425183-agent-certs\") pod \"konnectivity-agent-9g9nb\" (UID: \"1ed41459-ca2d-4277-a2df-c7879f425183\") " pod="kube-system/konnectivity-agent-9g9nb" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a73b204-193b-48be-9f93-a6cf774ec172-ovn-node-metrics-cert\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.688133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b7ee997-0723-4a47-90b5-805cd7721382-dbus\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:32.688948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-var-lib-kubelet\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.688948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687055 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a73b204-193b-48be-9f93-a6cf774ec172-ovnkube-script-lib\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.688948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.688948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-registration-dir\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.688948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-var-lib-cni-bin\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.688948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-lib-modules\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.688948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687164 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b9c86ed-68b3-4864-9824-8e55025d48ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.688948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5114137-2635-4a57-bdae-7866a1b417a2-tmp-dir\") pod \"node-resolver-95fpk\" (UID: \"d5114137-2635-4a57-bdae-7866a1b417a2\") " pod="openshift-dns/node-resolver-95fpk" Sep 08 10:01:32.688948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-var-lib-cni-bin\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.688948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-sys-fs\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.688948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-run-multus-certs\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.688948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687260 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b9c86ed-68b3-4864-9824-8e55025d48ba-system-cni-dir\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.688948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687284 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce8608f0-a21a-45fb-affd-0857e3802aca-host-run-multus-certs\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.688948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687321 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b9c86ed-68b3-4864-9824-8e55025d48ba-system-cni-dir\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.688948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.687664 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 08 10:01:32.694393 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.694371 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czmhr\" (UniqueName: \"kubernetes.io/projected/3b9c86ed-68b3-4864-9824-8e55025d48ba-kube-api-access-czmhr\") pod \"multus-additional-cni-plugins-5mv87\" (UID: \"3b9c86ed-68b3-4864-9824-8e55025d48ba\") " pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.694838 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.694821 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn2gw\" (UniqueName: \"kubernetes.io/projected/01480054-d16a-4820-af73-7a493b53ead0-kube-api-access-xn2gw\") pod \"network-metrics-daemon-jgstq\" (UID: \"01480054-d16a-4820-af73-7a493b53ead0\") " pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:32.695540 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.695511 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nbww\" (UniqueName: \"kubernetes.io/projected/aa2fe904-ae11-4f90-b530-f9bbcf80937d-kube-api-access-2nbww\") pod \"iptables-alerter-qhsv4\" (UID: \"aa2fe904-ae11-4f90-b530-f9bbcf80937d\") " pod="openshift-network-operator/iptables-alerter-qhsv4" Sep 08 10:01:32.696517 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.696497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m66r8\" (UniqueName: \"kubernetes.io/projected/1a73b204-193b-48be-9f93-a6cf774ec172-kube-api-access-m66r8\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.696701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.696662 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a73b204-193b-48be-9f93-a6cf774ec172-ovn-node-metrics-cert\") pod \"ovnkube-node-4fcgv\" (UID: \"1a73b204-193b-48be-9f93-a6cf774ec172\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.698084 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.698061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grfx9\" (UniqueName: \"kubernetes.io/projected/ce8608f0-a21a-45fb-affd-0857e3802aca-kube-api-access-grfx9\") pod \"multus-28d92\" (UID: \"ce8608f0-a21a-45fb-affd-0857e3802aca\") " pod="openshift-multus/multus-28d92" Sep 08 10:01:32.787770 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.787741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ed41459-ca2d-4277-a2df-c7879f425183-agent-certs\") pod \"konnectivity-agent-9g9nb\" (UID: \"1ed41459-ca2d-4277-a2df-c7879f425183\") " pod="kube-system/konnectivity-agent-9g9nb" Sep 08 10:01:32.787934 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.787786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b7ee997-0723-4a47-90b5-805cd7721382-dbus\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:32.787934 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.787850 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b7ee997-0723-4a47-90b5-805cd7721382-dbus\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:32.787934 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.787883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.787934 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.787904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-registration-dir\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.787934 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.787919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-lib-modules\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.787934 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.787936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5114137-2635-4a57-bdae-7866a1b417a2-tmp-dir\") pod \"node-resolver-95fpk\" (UID: \"d5114137-2635-4a57-bdae-7866a1b417a2\") " pod="openshift-dns/node-resolver-95fpk" Sep 08 10:01:32.788211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.787957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-sys-fs\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.788211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.787979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-registration-dir\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.788211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.787985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dkq8\" (UniqueName: \"kubernetes.io/projected/02dac30c-8810-4609-8768-4047fd4405da-kube-api-access-5dkq8\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.788211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788021 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-run\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.788211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788033 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-lib-modules\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.788211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12807ade-0ca9-4bd3-b277-afba07e51a26-host\") pod \"node-ca-wqx5v\" (UID: \"12807ade-0ca9-4bd3-b277-afba07e51a26\") " pod="openshift-image-registry/node-ca-wqx5v" Sep 08 10:01:32.788211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788060 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-sys-fs\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.788211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788074 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12807ade-0ca9-4bd3-b277-afba07e51a26-host\") pod \"node-ca-wqx5v\" (UID: \"12807ade-0ca9-4bd3-b277-afba07e51a26\") " pod="openshift-image-registry/node-ca-wqx5v" Sep 08 10:01:32.788211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788086 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-var-lib-kubelet\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.788211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-host\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.788211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-kubernetes\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.788211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d180476a-ce29-4508-80fe-3151edbba85e-etc-tuned\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.788211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788174 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-run\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.788211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d180476a-ce29-4508-80fe-3151edbba85e-tmp\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.788211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d5114137-2635-4a57-bdae-7866a1b417a2-hosts-file\") pod \"node-resolver-95fpk\" (UID: \"d5114137-2635-4a57-bdae-7866a1b417a2\") " pod="openshift-dns/node-resolver-95fpk" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b95h\" (UniqueName: \"kubernetes.io/projected/d5114137-2635-4a57-bdae-7866a1b417a2-kube-api-access-7b95h\") pod \"node-resolver-95fpk\" (UID: \"d5114137-2635-4a57-bdae-7866a1b417a2\") " pod="openshift-dns/node-resolver-95fpk" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9nw\" (UniqueName: \"kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw\") pod \"network-check-target-mvf7l\" (UID: \"334662a7-e44f-4aa9-8304-0a13dfed33e7\") " pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-socket-dir\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-sys\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788320 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5114137-2635-4a57-bdae-7866a1b417a2-tmp-dir\") pod \"node-resolver-95fpk\" (UID: \"d5114137-2635-4a57-bdae-7866a1b417a2\") " pod="openshift-dns/node-resolver-95fpk" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788356 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-host\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788377 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b7ee997-0723-4a47-90b5-805cd7721382-kubelet-config\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788398 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-var-lib-kubelet\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788427 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d5114137-2635-4a57-bdae-7866a1b417a2-hosts-file\") pod \"node-resolver-95fpk\" (UID: \"d5114137-2635-4a57-bdae-7866a1b417a2\") " pod="openshift-dns/node-resolver-95fpk" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788475 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-sys\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788481 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-socket-dir\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b7ee997-0723-4a47-90b5-805cd7721382-kubelet-config\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788520 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-modprobe-d\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788532 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-kubernetes\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-sysctl-d\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.788655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9dpn\" (UniqueName: \"kubernetes.io/projected/12807ade-0ca9-4bd3-b277-afba07e51a26-kube-api-access-c9dpn\") pod \"node-ca-wqx5v\" (UID: \"12807ade-0ca9-4bd3-b277-afba07e51a26\") " pod="openshift-image-registry/node-ca-wqx5v" Sep 08 10:01:32.789169 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.788597 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-sysctl-conf\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.792774 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.792749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5pwp\" (UniqueName: \"kubernetes.io/projected/d180476a-ce29-4508-80fe-3151edbba85e-kube-api-access-r5pwp\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.792894 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.792805 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-systemd\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.792894 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.792835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:32.792894 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.792862 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12807ade-0ca9-4bd3-b277-afba07e51a26-serviceca\") pod \"node-ca-wqx5v\" (UID: \"12807ade-0ca9-4bd3-b277-afba07e51a26\") " pod="openshift-image-registry/node-ca-wqx5v" Sep 08 10:01:32.792894 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.792885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-device-dir\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.793075 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.792912 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-etc-selinux\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.793075 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.792947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-sysconfig\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.793075 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.792974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ed41459-ca2d-4277-a2df-c7879f425183-konnectivity-ca\") pod \"konnectivity-agent-9g9nb\" (UID: \"1ed41459-ca2d-4277-a2df-c7879f425183\") " pod="kube-system/konnectivity-agent-9g9nb" Sep 08 10:01:32.793442 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.793424 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ed41459-ca2d-4277-a2df-c7879f425183-konnectivity-ca\") pod \"konnectivity-agent-9g9nb\" (UID: \"1ed41459-ca2d-4277-a2df-c7879f425183\") " pod="kube-system/konnectivity-agent-9g9nb" Sep 08 10:01:32.793725 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.793706 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-systemd\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.793809 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:32.793796 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 10:01:32.793869 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:32.793859 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret podName:2b7ee997-0723-4a47-90b5-805cd7721382 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:33.293839229 +0000 UTC m=+3.082891023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret") pod "global-pull-secret-syncer-ktdpl" (UID: "2b7ee997-0723-4a47-90b5-805cd7721382") : object "kube-system"/"original-pull-secret" not registered Sep 08 10:01:32.794997 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.794484 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ed41459-ca2d-4277-a2df-c7879f425183-agent-certs\") pod \"konnectivity-agent-9g9nb\" (UID: \"1ed41459-ca2d-4277-a2df-c7879f425183\") " pod="kube-system/konnectivity-agent-9g9nb" Sep 08 10:01:32.794997 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.794648 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-sysctl-d\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.794997 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.794664 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d180476a-ce29-4508-80fe-3151edbba85e-tmp\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.794997 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.794763 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-modprobe-d\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.794997 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.794821 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-device-dir\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.794997 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.794827 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02dac30c-8810-4609-8768-4047fd4405da-etc-selinux\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.794997 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.794831 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d180476a-ce29-4508-80fe-3151edbba85e-etc-tuned\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.794997 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.794887 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-sysconfig\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.795370 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.795108 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d180476a-ce29-4508-80fe-3151edbba85e-etc-sysctl-conf\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.795370 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.795219 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12807ade-0ca9-4bd3-b277-afba07e51a26-serviceca\") pod \"node-ca-wqx5v\" (UID: \"12807ade-0ca9-4bd3-b277-afba07e51a26\") " pod="openshift-image-registry/node-ca-wqx5v" Sep 08 10:01:32.797280 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.797257 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dkq8\" (UniqueName: \"kubernetes.io/projected/02dac30c-8810-4609-8768-4047fd4405da-kube-api-access-5dkq8\") pod \"aws-ebs-csi-driver-node-c98x4\" (UID: \"02dac30c-8810-4609-8768-4047fd4405da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:32.798480 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.798445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b95h\" (UniqueName: \"kubernetes.io/projected/d5114137-2635-4a57-bdae-7866a1b417a2-kube-api-access-7b95h\") pod \"node-resolver-95fpk\" (UID: \"d5114137-2635-4a57-bdae-7866a1b417a2\") " pod="openshift-dns/node-resolver-95fpk" Sep 08 10:01:32.799998 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:32.799980 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 10:01:32.800136 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:32.800001 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 10:01:32.800136 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:32.800013 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cq9nw for pod openshift-network-diagnostics/network-check-target-mvf7l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 10:01:32.800136 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:32.800067 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw podName:334662a7-e44f-4aa9-8304-0a13dfed33e7 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:33.300051097 +0000 UTC m=+3.089102897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cq9nw" (UniqueName: "kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw") pod "network-check-target-mvf7l" (UID: "334662a7-e44f-4aa9-8304-0a13dfed33e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 10:01:32.801576 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.801558 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5pwp\" (UniqueName: \"kubernetes.io/projected/d180476a-ce29-4508-80fe-3151edbba85e-kube-api-access-r5pwp\") pod \"tuned-lsd8z\" (UID: \"d180476a-ce29-4508-80fe-3151edbba85e\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.802430 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.802394 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9dpn\" (UniqueName: \"kubernetes.io/projected/12807ade-0ca9-4bd3-b277-afba07e51a26-kube-api-access-c9dpn\") pod \"node-ca-wqx5v\" (UID: \"12807ade-0ca9-4bd3-b277-afba07e51a26\") " pod="openshift-image-registry/node-ca-wqx5v" Sep 08 10:01:32.811734 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.811687 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 10:01:32.875540 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.875512 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:32.883018 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.882993 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-28d92" Sep 08 10:01:32.891845 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.891823 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qhsv4" Sep 08 10:01:32.896685 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.896661 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5mv87" Sep 08 10:01:32.903123 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.903075 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" Sep 08 10:01:32.910691 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.910519 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9g9nb" Sep 08 10:01:32.919865 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.919845 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-95fpk" Sep 08 10:01:32.925359 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.925344 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wqx5v" Sep 08 10:01:32.931796 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:32.931774 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" Sep 08 10:01:33.195754 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:33.195684 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs\") pod \"network-metrics-daemon-jgstq\" (UID: \"01480054-d16a-4820-af73-7a493b53ead0\") " pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:33.195894 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:33.195803 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 10:01:33.195894 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:33.195862 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs podName:01480054-d16a-4820-af73-7a493b53ead0 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:34.195844011 +0000 UTC m=+3.984895790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs") pod "network-metrics-daemon-jgstq" (UID: "01480054-d16a-4820-af73-7a493b53ead0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 10:01:33.296458 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:33.296433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:33.296645 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:33.296529 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 10:01:33.296645 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:33.296579 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret podName:2b7ee997-0723-4a47-90b5-805cd7721382 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:34.296563053 +0000 UTC m=+4.085614850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret") pod "global-pull-secret-syncer-ktdpl" (UID: "2b7ee997-0723-4a47-90b5-805cd7721382") : object "kube-system"/"original-pull-secret" not registered Sep 08 10:01:33.396816 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:33.396783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9nw\" (UniqueName: \"kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw\") pod \"network-check-target-mvf7l\" (UID: \"334662a7-e44f-4aa9-8304-0a13dfed33e7\") " pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:33.396981 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:33.396909 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 10:01:33.396981 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:33.396930 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 10:01:33.396981 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:33.396942 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cq9nw for pod openshift-network-diagnostics/network-check-target-mvf7l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 10:01:33.397143 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:33.396998 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw podName:334662a7-e44f-4aa9-8304-0a13dfed33e7 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:34.396981908 +0000 UTC m=+4.186033701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cq9nw" (UniqueName: "kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw") pod "network-check-target-mvf7l" (UID: "334662a7-e44f-4aa9-8304-0a13dfed33e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 10:01:33.606659 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:33.606538 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 09:56:32 +0000 UTC" deadline="2028-03-08 23:25:37.693238821 +0000 UTC" Sep 08 10:01:33.606659 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:33.606593 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13141h24m4.086648335s" Sep 08 10:01:33.671998 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:33.671960 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:33.672152 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:33.671971 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:33.672152 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:33.672083 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:01:33.672255 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:33.672159 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:01:33.706537 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:33.706506 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 10:01:33.822086 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:33.821914 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a73b204_193b_48be_9f93_a6cf774ec172.slice/crio-7e0db0bafcfd90820de4a14984b7834c709395ac491e4a06ea68066df50fdd74 WatchSource:0}: Error finding container 7e0db0bafcfd90820de4a14984b7834c709395ac491e4a06ea68066df50fdd74: Status 404 returned error can't find the container with id 7e0db0bafcfd90820de4a14984b7834c709395ac491e4a06ea68066df50fdd74 Sep 08 10:01:33.824429 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:33.824406 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12807ade_0ca9_4bd3_b277_afba07e51a26.slice/crio-47a44722914322c46524e11963ae445f1a376691e6552f586e28099e1d4a8afc WatchSource:0}: Error finding container 47a44722914322c46524e11963ae445f1a376691e6552f586e28099e1d4a8afc: Status 404 returned error can't find the container with id 47a44722914322c46524e11963ae445f1a376691e6552f586e28099e1d4a8afc Sep 08 10:01:33.846044 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:33.846021 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd180476a_ce29_4508_80fe_3151edbba85e.slice/crio-29ea61bb16cc4767a9b16477cec0a44d0e8de95deef20ad0e3dda61152ad3ea6 WatchSource:0}: Error finding container 29ea61bb16cc4767a9b16477cec0a44d0e8de95deef20ad0e3dda61152ad3ea6: Status 404 returned error can't find the container with id 29ea61bb16cc4767a9b16477cec0a44d0e8de95deef20ad0e3dda61152ad3ea6 Sep 08 10:01:33.850502 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:33.850483 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9c86ed_68b3_4864_9824_8e55025d48ba.slice/crio-7eb13a142ee9da12a08b49ae32a2e847d00d63dde654ba18eb1f100b827d1041 WatchSource:0}: Error finding container 7eb13a142ee9da12a08b49ae32a2e847d00d63dde654ba18eb1f100b827d1041: Status 404 returned error can't find the container with id 7eb13a142ee9da12a08b49ae32a2e847d00d63dde654ba18eb1f100b827d1041 Sep 08 10:01:33.850721 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:33.850703 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5114137_2635_4a57_bdae_7866a1b417a2.slice/crio-b58afd386dc90991d72115b75e48794c2d7fd61bf0af4c47b8acc106829db5a6 WatchSource:0}: Error finding container b58afd386dc90991d72115b75e48794c2d7fd61bf0af4c47b8acc106829db5a6: Status 404 returned error can't find the container with id b58afd386dc90991d72115b75e48794c2d7fd61bf0af4c47b8acc106829db5a6 Sep 08 10:01:33.851698 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:33.851642 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02dac30c_8810_4609_8768_4047fd4405da.slice/crio-15fcd5c4e1afc8e37c266ddb97441cb2c0fcf1153733bfe620890935be2a61ab WatchSource:0}: Error finding container 15fcd5c4e1afc8e37c266ddb97441cb2c0fcf1153733bfe620890935be2a61ab: Status 404 returned error can't find the container with id 15fcd5c4e1afc8e37c266ddb97441cb2c0fcf1153733bfe620890935be2a61ab Sep 08 10:01:33.852453 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:33.852432 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2fe904_ae11_4f90_b530_f9bbcf80937d.slice/crio-9c9dead1358951271e1ba537acbb08e5bc53eab3893cfc046112f184bd875bf6 WatchSource:0}: Error finding container 9c9dead1358951271e1ba537acbb08e5bc53eab3893cfc046112f184bd875bf6: Status 404 returned error can't find the container with id 9c9dead1358951271e1ba537acbb08e5bc53eab3893cfc046112f184bd875bf6 Sep 08 10:01:33.853316 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:33.853279 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8608f0_a21a_45fb_affd_0857e3802aca.slice/crio-6b0ee3b71e5e2708d8837cef142f6b61dd8d537ea657654d846265df4b0bd7c2 WatchSource:0}: Error finding container 6b0ee3b71e5e2708d8837cef142f6b61dd8d537ea657654d846265df4b0bd7c2: Status 404 returned error can't find the container with id 6b0ee3b71e5e2708d8837cef142f6b61dd8d537ea657654d846265df4b0bd7c2 Sep 08 10:01:33.854835 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:01:33.854745 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed41459_ca2d_4277_a2df_c7879f425183.slice/crio-17dc7734f1f1962b19295cc5417369795ced1574a67227c82a5c59236070cf8a WatchSource:0}: Error finding container 17dc7734f1f1962b19295cc5417369795ced1574a67227c82a5c59236070cf8a: Status 404 returned error can't find the container with id 17dc7734f1f1962b19295cc5417369795ced1574a67227c82a5c59236070cf8a Sep 08 10:01:34.203136 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.203046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs\") pod \"network-metrics-daemon-jgstq\" (UID: \"01480054-d16a-4820-af73-7a493b53ead0\") " pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:34.203269 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:34.203186 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 10:01:34.203269 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:34.203249 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs podName:01480054-d16a-4820-af73-7a493b53ead0 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:36.203234533 +0000 UTC m=+5.992286317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs") pod "network-metrics-daemon-jgstq" (UID: "01480054-d16a-4820-af73-7a493b53ead0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 10:01:34.304555 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.304007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:34.304555 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:34.304157 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 10:01:34.304555 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:34.304221 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret podName:2b7ee997-0723-4a47-90b5-805cd7721382 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:36.304201489 +0000 UTC m=+6.093253293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret") pod "global-pull-secret-syncer-ktdpl" (UID: "2b7ee997-0723-4a47-90b5-805cd7721382") : object "kube-system"/"original-pull-secret" not registered Sep 08 10:01:34.405201 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.404586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9nw\" (UniqueName: \"kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw\") pod \"network-check-target-mvf7l\" (UID: \"334662a7-e44f-4aa9-8304-0a13dfed33e7\") " pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:34.405201 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:34.404782 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 10:01:34.405201 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:34.404802 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 10:01:34.405201 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:34.404814 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cq9nw for pod openshift-network-diagnostics/network-check-target-mvf7l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 10:01:34.405201 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:34.404868 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw podName:334662a7-e44f-4aa9-8304-0a13dfed33e7 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:36.404852769 +0000 UTC m=+6.193904562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cq9nw" (UniqueName: "kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw") pod "network-check-target-mvf7l" (UID: "334662a7-e44f-4aa9-8304-0a13dfed33e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 10:01:34.607696 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.607563 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 09:56:32 +0000 UTC" deadline="2028-03-01 20:43:32.616144155 +0000 UTC" Sep 08 10:01:34.607696 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.607609 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12970h41m58.008547355s" Sep 08 10:01:34.674291 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.674264 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:34.674451 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:34.674375 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:01:34.680719 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.680691 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-28d92" event={"ID":"ce8608f0-a21a-45fb-affd-0857e3802aca","Type":"ContainerStarted","Data":"6b0ee3b71e5e2708d8837cef142f6b61dd8d537ea657654d846265df4b0bd7c2"} Sep 08 10:01:34.682649 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.682603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-95fpk" event={"ID":"d5114137-2635-4a57-bdae-7866a1b417a2","Type":"ContainerStarted","Data":"b58afd386dc90991d72115b75e48794c2d7fd61bf0af4c47b8acc106829db5a6"} Sep 08 10:01:34.684446 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.684421 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wqx5v" event={"ID":"12807ade-0ca9-4bd3-b277-afba07e51a26","Type":"ContainerStarted","Data":"47a44722914322c46524e11963ae445f1a376691e6552f586e28099e1d4a8afc"} Sep 08 10:01:34.688545 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.688522 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" event={"ID":"1a73b204-193b-48be-9f93-a6cf774ec172","Type":"ContainerStarted","Data":"7e0db0bafcfd90820de4a14984b7834c709395ac491e4a06ea68066df50fdd74"} Sep 08 10:01:34.691280 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.691257 2576 generic.go:356] "Generic (PLEG): container finished" podID="19e5ada69ca2575a19d6db1fc5d72bd7" containerID="1707d58cf3c4d9dd4b4cc80964d555631e5f180d2d1e5ba2f54a84a6d894580f" exitCode=0 Sep 08 10:01:34.691365 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.691327 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal" event={"ID":"19e5ada69ca2575a19d6db1fc5d72bd7","Type":"ContainerDied","Data":"1707d58cf3c4d9dd4b4cc80964d555631e5f180d2d1e5ba2f54a84a6d894580f"} Sep 08 10:01:34.697122 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.696132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-206.ec2.internal" event={"ID":"aee6609005f8620b26930034856bde0d","Type":"ContainerStarted","Data":"fd408d165e1972b4b93a82033d888a4a36b62b300d020377dc54b0752acd7d0c"} Sep 08 10:01:34.699164 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.699141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9g9nb" event={"ID":"1ed41459-ca2d-4277-a2df-c7879f425183","Type":"ContainerStarted","Data":"17dc7734f1f1962b19295cc5417369795ced1574a67227c82a5c59236070cf8a"} Sep 08 10:01:34.702023 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.702000 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qhsv4" event={"ID":"aa2fe904-ae11-4f90-b530-f9bbcf80937d","Type":"ContainerStarted","Data":"9c9dead1358951271e1ba537acbb08e5bc53eab3893cfc046112f184bd875bf6"} Sep 08 10:01:34.703812 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.703789 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" event={"ID":"02dac30c-8810-4609-8768-4047fd4405da","Type":"ContainerStarted","Data":"15fcd5c4e1afc8e37c266ddb97441cb2c0fcf1153733bfe620890935be2a61ab"} Sep 08 10:01:34.716636 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.711665 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mv87" event={"ID":"3b9c86ed-68b3-4864-9824-8e55025d48ba","Type":"ContainerStarted","Data":"7eb13a142ee9da12a08b49ae32a2e847d00d63dde654ba18eb1f100b827d1041"} Sep 08 10:01:34.716636 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.716434 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" event={"ID":"d180476a-ce29-4508-80fe-3151edbba85e","Type":"ContainerStarted","Data":"29ea61bb16cc4767a9b16477cec0a44d0e8de95deef20ad0e3dda61152ad3ea6"} Sep 08 10:01:34.726050 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:34.725998 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-206.ec2.internal" podStartSLOduration=3.725982028 podStartE2EDuration="3.725982028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:34.725903218 +0000 UTC m=+4.514955020" watchObservedRunningTime="2026-09-08 10:01:34.725982028 +0000 UTC m=+4.515033829" Sep 08 10:01:35.672766 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:35.672142 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:35.672766 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:35.672265 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:01:35.672766 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:35.672625 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:35.672766 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:35.672719 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:01:35.727263 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:35.727226 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal" event={"ID":"19e5ada69ca2575a19d6db1fc5d72bd7","Type":"ContainerStarted","Data":"b0d2d6e8afe84ae80ca7d4a80a8fdc10c0c4ae494d59b12f97071ea6c4f1a35e"} Sep 08 10:01:35.741244 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:35.741192 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-206.ec2.internal" podStartSLOduration=4.741172849 podStartE2EDuration="4.741172849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:35.740648653 +0000 UTC m=+5.529700459" watchObservedRunningTime="2026-09-08 10:01:35.741172849 +0000 UTC m=+5.530224654" Sep 08 10:01:36.222930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:36.222880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs\") pod \"network-metrics-daemon-jgstq\" (UID: \"01480054-d16a-4820-af73-7a493b53ead0\") " pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:36.223110 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:36.223092 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 10:01:36.223191 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:36.223168 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs podName:01480054-d16a-4820-af73-7a493b53ead0 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:40.223149634 +0000 UTC m=+10.012201419 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs") pod "network-metrics-daemon-jgstq" (UID: "01480054-d16a-4820-af73-7a493b53ead0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 10:01:36.323585 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:36.323543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:36.323782 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:36.323712 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 10:01:36.323841 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:36.323795 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret podName:2b7ee997-0723-4a47-90b5-805cd7721382 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:40.323774871 +0000 UTC m=+10.112826656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret") pod "global-pull-secret-syncer-ktdpl" (UID: "2b7ee997-0723-4a47-90b5-805cd7721382") : object "kube-system"/"original-pull-secret" not registered Sep 08 10:01:36.424276 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:36.424244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9nw\" (UniqueName: \"kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw\") pod \"network-check-target-mvf7l\" (UID: \"334662a7-e44f-4aa9-8304-0a13dfed33e7\") " pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:36.424521 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:36.424392 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 10:01:36.424521 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:36.424411 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 10:01:36.424521 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:36.424422 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cq9nw for pod openshift-network-diagnostics/network-check-target-mvf7l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 10:01:36.424521 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:36.424487 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw podName:334662a7-e44f-4aa9-8304-0a13dfed33e7 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:40.424468856 +0000 UTC m=+10.213520657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cq9nw" (UniqueName: "kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw") pod "network-check-target-mvf7l" (UID: "334662a7-e44f-4aa9-8304-0a13dfed33e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 10:01:36.671753 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:36.671668 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:36.671917 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:36.671806 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:01:37.671322 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:37.671289 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:37.671778 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:37.671397 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:01:37.671778 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:37.671428 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:37.671778 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:37.671483 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:01:38.671729 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:38.671691 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:38.672184 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:38.671832 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:01:39.672144 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:39.672061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:39.672144 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:39.672087 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:39.672639 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:39.672195 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:01:39.672639 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:39.672328 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:01:40.252913 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:40.252766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs\") pod \"network-metrics-daemon-jgstq\" (UID: \"01480054-d16a-4820-af73-7a493b53ead0\") " pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:40.253097 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:40.252947 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 10:01:40.253097 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:40.253018 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs podName:01480054-d16a-4820-af73-7a493b53ead0 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:48.253000048 +0000 UTC m=+18.042051839 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs") pod "network-metrics-daemon-jgstq" (UID: "01480054-d16a-4820-af73-7a493b53ead0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 10:01:40.354018 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:40.353983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:40.354183 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:40.354131 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 10:01:40.354256 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:40.354204 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret podName:2b7ee997-0723-4a47-90b5-805cd7721382 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:48.354184363 +0000 UTC m=+18.143236161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret") pod "global-pull-secret-syncer-ktdpl" (UID: "2b7ee997-0723-4a47-90b5-805cd7721382") : object "kube-system"/"original-pull-secret" not registered Sep 08 10:01:40.455197 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:40.455079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9nw\" (UniqueName: \"kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw\") pod \"network-check-target-mvf7l\" (UID: \"334662a7-e44f-4aa9-8304-0a13dfed33e7\") " pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:40.455367 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:40.455262 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 10:01:40.455367 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:40.455294 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 10:01:40.455367 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:40.455307 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cq9nw for pod openshift-network-diagnostics/network-check-target-mvf7l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 10:01:40.455367 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:40.455364 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw podName:334662a7-e44f-4aa9-8304-0a13dfed33e7 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:48.45534519 +0000 UTC m=+18.244396974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cq9nw" (UniqueName: "kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw") pod "network-check-target-mvf7l" (UID: "334662a7-e44f-4aa9-8304-0a13dfed33e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 10:01:40.672313 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:40.672234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:40.672690 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:40.672374 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:01:41.671769 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:41.671502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:41.671957 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:41.671534 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:41.671957 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:41.671881 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:01:41.672070 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:41.671965 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:01:41.742805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:41.742724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9g9nb" event={"ID":"1ed41459-ca2d-4277-a2df-c7879f425183","Type":"ContainerStarted","Data":"191bed5e68476193173f4b37bac25437223b754f980c47d56be8b71099d0205c"} Sep 08 10:01:41.745414 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:41.745388 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" event={"ID":"02dac30c-8810-4609-8768-4047fd4405da","Type":"ContainerStarted","Data":"da0a58052a88c7c315a732b0ccea43c0256fa78345b3c769a9c6aff8cf44b61b"} Sep 08 10:01:41.747699 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:41.747673 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b9c86ed-68b3-4864-9824-8e55025d48ba" containerID="e6967492dcdb2eef22bde7f8ff3c2292287f9512b05d86a3251945230a5aecd1" exitCode=0 Sep 08 10:01:41.747818 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:41.747754 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mv87" event={"ID":"3b9c86ed-68b3-4864-9824-8e55025d48ba","Type":"ContainerDied","Data":"e6967492dcdb2eef22bde7f8ff3c2292287f9512b05d86a3251945230a5aecd1"} Sep 08 10:01:41.750574 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:41.750544 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" event={"ID":"d180476a-ce29-4508-80fe-3151edbba85e","Type":"ContainerStarted","Data":"358b663e7e7685599b4a9c7a417a809fd0880cf098b7ad4635a0945cd4ea9649"} Sep 08 10:01:41.753578 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:41.753553 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-95fpk" event={"ID":"d5114137-2635-4a57-bdae-7866a1b417a2","Type":"ContainerStarted","Data":"ac4bb4708b0c574ba97a40070243f93943f5ef76e5033362dac9863fcf461e33"} Sep 08 10:01:41.756011 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:41.755980 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wqx5v" event={"ID":"12807ade-0ca9-4bd3-b277-afba07e51a26","Type":"ContainerStarted","Data":"18159365a8fbdc2491225fd65ce97d672f839fad07a684d13ad6d285a14f9a01"} Sep 08 10:01:41.771153 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:41.771096 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9g9nb" podStartSLOduration=4.395076971 podStartE2EDuration="11.771075992s" totalImagesPullingTime="7.375999021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:41.770528548 +0000 UTC m=+11.559580364" watchObservedRunningTime="2026-09-08 10:01:41.771075992 +0000 UTC m=+11.560127798" Sep 08 10:01:41.790740 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:41.790452 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lsd8z" podStartSLOduration=4.371612296 podStartE2EDuration="11.790433522s" totalImagesPullingTime="7.418821226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:41.789921276 +0000 UTC m=+11.578973080" watchObservedRunningTime="2026-09-08 10:01:41.790433522 +0000 UTC m=+11.579485328" Sep 08 10:01:41.813863 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:41.813810 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-95fpk" podStartSLOduration=4.434225874 podStartE2EDuration="11.813797177s" totalImagesPullingTime="7.379571303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:41.813586137 +0000 UTC m=+11.602637941" watchObservedRunningTime="2026-09-08 10:01:41.813797177 +0000 UTC m=+11.602848984" Sep 08 10:01:41.827472 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:41.827423 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wqx5v" podStartSLOduration=4.455346988 podStartE2EDuration="11.827407639s" totalImagesPullingTime="7.372060651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:41.826916018 +0000 UTC m=+11.615967823" watchObservedRunningTime="2026-09-08 10:01:41.827407639 +0000 UTC m=+11.616459443" Sep 08 10:01:42.345952 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:42.345908 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9g9nb" Sep 08 10:01:42.346780 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:42.346756 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9g9nb" Sep 08 10:01:42.672539 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:42.672504 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:42.672720 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:42.672648 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:01:43.671527 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:43.671495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:43.672050 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:43.671496 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:43.672050 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:43.671667 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:01:43.672050 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:43.671808 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:01:43.762426 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:43.762387 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qhsv4" event={"ID":"aa2fe904-ae11-4f90-b530-f9bbcf80937d","Type":"ContainerStarted","Data":"33810a225904aca04257ce871be1cf953fce13bb461bd769f60db8ba0941c69d"} Sep 08 10:01:43.762426 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:43.762422 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 08 10:01:43.777145 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:43.777061 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qhsv4" podStartSLOduration=6.399144297 podStartE2EDuration="13.777040213s" totalImagesPullingTime="7.377895916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:43.776964489 +0000 UTC m=+13.566016294" watchObservedRunningTime="2026-09-08 10:01:43.777040213 +0000 UTC m=+13.566092018" Sep 08 10:01:44.672145 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:44.672113 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:44.672609 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:44.672221 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:01:45.671516 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:45.671485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:45.671726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:45.671485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:45.671726 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:45.671640 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:01:45.671834 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:45.671775 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:01:46.671185 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:46.671145 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:46.671735 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:46.671256 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:01:47.671781 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:47.671737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:47.671781 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:47.671779 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:47.672334 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:47.671923 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:01:47.672334 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:47.672058 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:01:48.321579 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:48.321543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs\") pod \"network-metrics-daemon-jgstq\" (UID: \"01480054-d16a-4820-af73-7a493b53ead0\") " pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:48.321759 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:48.321727 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 10:01:48.321821 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:48.321810 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs podName:01480054-d16a-4820-af73-7a493b53ead0 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:04.321787541 +0000 UTC m=+34.110839338 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs") pod "network-metrics-daemon-jgstq" (UID: "01480054-d16a-4820-af73-7a493b53ead0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 10:01:48.422883 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:48.422846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:48.423063 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:48.423021 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 10:01:48.423106 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:48.423096 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret podName:2b7ee997-0723-4a47-90b5-805cd7721382 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:04.423077809 +0000 UTC m=+34.212129592 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret") pod "global-pull-secret-syncer-ktdpl" (UID: "2b7ee997-0723-4a47-90b5-805cd7721382") : object "kube-system"/"original-pull-secret" not registered Sep 08 10:01:48.523588 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:48.523549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9nw\" (UniqueName: \"kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw\") pod \"network-check-target-mvf7l\" (UID: \"334662a7-e44f-4aa9-8304-0a13dfed33e7\") " pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:48.523787 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:48.523763 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 10:01:48.523853 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:48.523796 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 10:01:48.523853 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:48.523810 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cq9nw for pod openshift-network-diagnostics/network-check-target-mvf7l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 10:01:48.523949 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:48.523876 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw podName:334662a7-e44f-4aa9-8304-0a13dfed33e7 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:04.523855842 +0000 UTC m=+34.312907631 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cq9nw" (UniqueName: "kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw") pod "network-check-target-mvf7l" (UID: "334662a7-e44f-4aa9-8304-0a13dfed33e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 10:01:48.671526 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:48.671436 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:48.671710 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:48.671581 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:01:49.272243 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:49.272195 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9g9nb" Sep 08 10:01:49.272764 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:49.272358 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 08 10:01:49.272907 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:49.272828 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9g9nb" Sep 08 10:01:49.671830 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:49.671745 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:49.671996 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:49.671873 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:01:49.671996 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:49.671894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:49.671996 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:49.671979 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:01:50.672572 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:50.672318 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:50.673054 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:50.672679 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:01:51.671558 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:51.671521 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:51.671743 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:51.671654 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:01:51.671743 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:51.671709 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:51.671852 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:51.671813 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:01:52.671920 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:52.671883 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:52.672358 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:52.672022 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:01:53.563978 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:53.563949 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 08 10:01:53.648009 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:53.647894 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T10:01:53.563973485Z","UUID":"2e11543b-c68e-4d41-9d90-4d26edd00bf5","Handler":null,"Name":"","Endpoint":""} Sep 08 10:01:53.650204 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:53.650152 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 08 10:01:53.650204 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:53.650187 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 08 10:01:53.671404 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:53.671119 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:53.671404 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:53.671119 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:53.671404 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:53.671230 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:01:53.671404 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:53.671312 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:01:53.785763 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:53.785719 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" event={"ID":"02dac30c-8810-4609-8768-4047fd4405da","Type":"ContainerStarted","Data":"91de5ef392c9b6bccf72eab43f6d4fd21c0ee9ae4137cf6f70aa4b8c9118e22a"} Sep 08 10:01:53.787879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:53.787845 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b9c86ed-68b3-4864-9824-8e55025d48ba" containerID="3fb3f2ec5cb356f2f2e1b797421d186987c4c86024418f79446b226a8478c40f" exitCode=0 Sep 08 10:01:53.787998 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:53.787924 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mv87" event={"ID":"3b9c86ed-68b3-4864-9824-8e55025d48ba","Type":"ContainerDied","Data":"3fb3f2ec5cb356f2f2e1b797421d186987c4c86024418f79446b226a8478c40f"} Sep 08 10:01:53.789522 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:53.789500 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-28d92" event={"ID":"ce8608f0-a21a-45fb-affd-0857e3802aca","Type":"ContainerStarted","Data":"783f2d0184c06f82ed7b9bb679784f40998884604ccffef21dd60dcfe480a4b5"} Sep 08 10:01:53.792573 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:53.792551 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" event={"ID":"1a73b204-193b-48be-9f93-a6cf774ec172","Type":"ContainerStarted","Data":"815018c8a23f28864992b8be06d8a34234eafdd68c91d555c91633924f1b848c"} Sep 08 10:01:53.792675 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:53.792605 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" event={"ID":"1a73b204-193b-48be-9f93-a6cf774ec172","Type":"ContainerStarted","Data":"3bcdf18b83393310f120de89c7af0b4f37923f492b361d060cebc3e56c4bbdb0"} Sep 08 10:01:53.792675 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:53.792654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" event={"ID":"1a73b204-193b-48be-9f93-a6cf774ec172","Type":"ContainerStarted","Data":"936825e488ca9b6b7a0656b762b4e7a9924de249fdfff3fa13ee53648333f200"} Sep 08 10:01:53.792756 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:53.792683 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" event={"ID":"1a73b204-193b-48be-9f93-a6cf774ec172","Type":"ContainerStarted","Data":"87bef4352dcd27b5ede0f828a5c2e0e78d03a59a8e690cf73b9a9fa04861340d"} Sep 08 10:01:53.822161 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:53.822107 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-28d92" podStartSLOduration=4.196195995 podStartE2EDuration="23.822085426s" totalImagesPullingTime="19.625889431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:53.821163705 +0000 UTC m=+23.610215511" watchObservedRunningTime="2026-09-08 10:01:53.822085426 +0000 UTC m=+23.611137231" Sep 08 10:01:54.671960 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:54.671935 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:54.672086 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:54.672061 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:01:54.797136 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:54.797103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" event={"ID":"1a73b204-193b-48be-9f93-a6cf774ec172","Type":"ContainerStarted","Data":"829249e0f5a42fc9797d598b4b3f72b908e3feffb5bb3e3bd1e8a47dff8d70ae"} Sep 08 10:01:54.797543 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:54.797184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" event={"ID":"1a73b204-193b-48be-9f93-a6cf774ec172","Type":"ContainerStarted","Data":"012a56baa1ce93c40d01bbb7785f7b3b60d483ca3dfeffffe6f29c238b6f829d"} Sep 08 10:01:54.799093 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:54.799065 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b9c86ed-68b3-4864-9824-8e55025d48ba" containerID="e50974c7d44b1a3446b173f8ed755621a8e75062f7d27c4f9d5d387507292971" exitCode=0 Sep 08 10:01:54.799252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:54.799150 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mv87" event={"ID":"3b9c86ed-68b3-4864-9824-8e55025d48ba","Type":"ContainerDied","Data":"e50974c7d44b1a3446b173f8ed755621a8e75062f7d27c4f9d5d387507292971"} Sep 08 10:01:55.276217 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:55.276188 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:01:55.357779 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:55.357742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:01:55.671916 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:55.671887 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:55.672049 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:55.671987 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:01:55.672049 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:55.672016 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:55.672123 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:55.672083 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:01:55.803984 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:55.803896 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b9c86ed-68b3-4864-9824-8e55025d48ba" containerID="49665934d8d3051a52a34321e915f89f6f68127a50176664a6c1792684baaf16" exitCode=0 Sep 08 10:01:55.803984 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:55.803933 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mv87" event={"ID":"3b9c86ed-68b3-4864-9824-8e55025d48ba","Type":"ContainerDied","Data":"49665934d8d3051a52a34321e915f89f6f68127a50176664a6c1792684baaf16"} Sep 08 10:01:56.671571 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:56.671529 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:56.671755 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:56.671686 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:01:56.743538 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:56.743506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:01:56.810206 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:56.810158 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" event={"ID":"1a73b204-193b-48be-9f93-a6cf774ec172","Type":"ContainerStarted","Data":"cf580a91a883363016554888b18b61855299b02035722af30610c881bcc9575a"} Sep 08 10:01:56.846084 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:56.846054 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:01:57.671773 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:57.671741 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:57.671960 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:57.671856 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:01:57.671960 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:57.671911 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:57.672074 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:57.672019 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:01:58.672425 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:58.672134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:01:58.672976 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:58.672859 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:01:58.819260 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:58.819168 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" event={"ID":"1a73b204-193b-48be-9f93-a6cf774ec172","Type":"ContainerStarted","Data":"0ef2f312968104ba413553683676f499392d9d470580fcab9c46a74d8bcff936"} Sep 08 10:01:58.819569 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:58.819543 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:58.819748 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:58.819707 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:58.821626 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:58.821601 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" event={"ID":"02dac30c-8810-4609-8768-4047fd4405da","Type":"ContainerStarted","Data":"8db6dc2f8b89738550e928695e6e1d54274ac2876e7a211208aaaae44599fcbd"} Sep 08 10:01:58.835316 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:58.835294 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:58.843854 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:58.843806 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" podStartSLOduration=9.212336757 podStartE2EDuration="28.843787634s" totalImagesPullingTime="19.631450877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:58.84364423 +0000 UTC m=+28.632696036" watchObservedRunningTime="2026-09-08 10:01:58.843787634 +0000 UTC m=+28.632839440" Sep 08 10:01:58.859775 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:58.859710 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c98x4" podStartSLOduration=4.634449331 podStartE2EDuration="28.859689385s" totalImagesPullingTime="24.225240054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:58.859083896 +0000 UTC m=+28.648135701" watchObservedRunningTime="2026-09-08 10:01:58.859689385 +0000 UTC m=+28.648741190" Sep 08 10:01:59.652574 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:59.652543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:01:59.671452 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:59.671418 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:01:59.671612 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:59.671549 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:01:59.671612 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:59.671557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:01:59.671725 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:01:59.671674 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:01:59.753735 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:59.753690 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:01:59.824532 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:59.824439 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:01:59.840998 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:01:59.840969 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:02:00.251130 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:00.251091 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jgstq"] Sep 08 10:02:00.251307 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:00.251213 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:02:00.251366 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:00.251315 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:02:00.254026 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:00.253991 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mvf7l"] Sep 08 10:02:00.254150 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:00.254105 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:02:00.254206 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:00.254182 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:02:00.254767 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:00.254709 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ktdpl"] Sep 08 10:02:00.254894 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:00.254793 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:02:00.254894 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:00.254874 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:02:01.671334 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:01.671294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:02:01.671911 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:01.671295 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:02:01.671911 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:01.671429 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:02:01.671911 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:01.671429 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:02:01.671911 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:01.671497 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:02:01.671911 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:01.671551 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:02:01.830786 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:01.830754 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b9c86ed-68b3-4864-9824-8e55025d48ba" containerID="8b210a5856a08bed3c98a66bd494766637d4ec238d1530ba99faabf267d30597" exitCode=0 Sep 08 10:02:01.830926 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:01.830800 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mv87" event={"ID":"3b9c86ed-68b3-4864-9824-8e55025d48ba","Type":"ContainerDied","Data":"8b210a5856a08bed3c98a66bd494766637d4ec238d1530ba99faabf267d30597"} Sep 08 10:02:02.837329 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:02.837294 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b9c86ed-68b3-4864-9824-8e55025d48ba" containerID="6bfd5d2220996c67a9acddb2f6c6eca438f400db4ce59be3309ac88b8ea14e0c" exitCode=0 Sep 08 10:02:02.837718 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:02.837352 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mv87" event={"ID":"3b9c86ed-68b3-4864-9824-8e55025d48ba","Type":"ContainerDied","Data":"6bfd5d2220996c67a9acddb2f6c6eca438f400db4ce59be3309ac88b8ea14e0c"} Sep 08 10:02:03.084664 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:03.084560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:03.144701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:03.144675 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:03.671837 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:03.671803 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:02:03.671837 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:03.671820 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:02:03.671837 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:03.671828 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:02:03.672113 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:03.671915 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:02:03.672113 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:03.672023 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:02:03.672113 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:03.672093 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:02:03.843373 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:03.843336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mv87" event={"ID":"3b9c86ed-68b3-4864-9824-8e55025d48ba","Type":"ContainerStarted","Data":"a26aab6ebd3d1e16ea0c0b18f349351445f74de5b92e177dccc8a65699434953"} Sep 08 10:02:03.865875 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:03.865822 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5mv87" podStartSLOduration=7.961859333 podStartE2EDuration="33.865803824s" totalImagesPullingTime="25.903944491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:03.865390626 +0000 UTC m=+33.654442430" watchObservedRunningTime="2026-09-08 10:02:03.865803824 +0000 UTC m=+33.654855627" Sep 08 10:02:03.987605 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:03.987523 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:04.044223 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:04.044172 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:04.106185 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:04.106148 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:04.291357 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:04.291272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:04.338727 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:04.338697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs\") pod \"network-metrics-daemon-jgstq\" (UID: \"01480054-d16a-4820-af73-7a493b53ead0\") " pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:02:04.338915 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:04.338842 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 10:02:04.338915 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:04.338907 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs podName:01480054-d16a-4820-af73-7a493b53ead0 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:36.338891569 +0000 UTC m=+66.127943351 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs") pod "network-metrics-daemon-jgstq" (UID: "01480054-d16a-4820-af73-7a493b53ead0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 10:02:04.357206 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:04.357160 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:04.439442 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:04.439410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:02:04.439666 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:04.439551 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 10:02:04.439666 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:04.439628 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret podName:2b7ee997-0723-4a47-90b5-805cd7721382 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:36.439600246 +0000 UTC m=+66.228652027 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret") pod "global-pull-secret-syncer-ktdpl" (UID: "2b7ee997-0723-4a47-90b5-805cd7721382") : object "kube-system"/"original-pull-secret" not registered Sep 08 10:02:04.539904 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:04.539866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9nw\" (UniqueName: \"kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw\") pod \"network-check-target-mvf7l\" (UID: \"334662a7-e44f-4aa9-8304-0a13dfed33e7\") " pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:02:04.540078 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:04.540029 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 10:02:04.540078 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:04.540048 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 10:02:04.540078 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:04.540057 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cq9nw for pod openshift-network-diagnostics/network-check-target-mvf7l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 10:02:04.540173 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:04.540110 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw podName:334662a7-e44f-4aa9-8304-0a13dfed33e7 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:36.540096918 +0000 UTC m=+66.329148700 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cq9nw" (UniqueName: "kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw") pod "network-check-target-mvf7l" (UID: "334662a7-e44f-4aa9-8304-0a13dfed33e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 10:02:04.814118 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:04.814081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:04.874936 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:04.874905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:05.671892 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:05.671857 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:02:05.671892 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:05.671895 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:02:05.672123 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:05.671895 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:02:05.672123 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:05.671971 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgstq" podUID="01480054-d16a-4820-af73-7a493b53ead0" Sep 08 10:02:05.672123 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:05.672066 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvf7l" podUID="334662a7-e44f-4aa9-8304-0a13dfed33e7" Sep 08 10:02:05.672232 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:05.672139 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktdpl" podUID="2b7ee997-0723-4a47-90b5-805cd7721382" Sep 08 10:02:05.998721 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:05.998645 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-206.ec2.internal" event="NodeReady" Sep 08 10:02:05.999062 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:05.998774 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 08 10:02:06.055458 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.055427 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n64nz"] Sep 08 10:02:06.074018 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.073989 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hqtdm"] Sep 08 10:02:06.074803 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.074195 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:06.077246 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.077220 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pd9dz\"" Sep 08 10:02:06.077378 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.077224 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 08 10:02:06.077378 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.077338 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 08 10:02:06.092440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.092417 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n64nz"] Sep 08 10:02:06.092549 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.092453 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d8gdx"] Sep 08 10:02:06.092549 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.092526 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.098337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.098314 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hrpwv\"" Sep 08 10:02:06.098337 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.098328 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 08 10:02:06.098526 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.098342 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 08 10:02:06.098526 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.098391 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 08 10:02:06.098526 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.098487 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 08 10:02:06.112930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.112899 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hqtdm"] Sep 08 10:02:06.113058 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.112953 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d8gdx"] Sep 08 10:02:06.113058 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.113005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d8gdx" Sep 08 10:02:06.117566 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.117547 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 08 10:02:06.117915 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.117899 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 08 10:02:06.117998 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.117900 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 08 10:02:06.117998 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.117945 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-94npr\"" Sep 08 10:02:06.151340 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.151300 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t25t8\" (UniqueName: \"kubernetes.io/projected/acdded14-6f01-4f17-97ea-7b9f7f1006eb-kube-api-access-t25t8\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:06.151527 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.151351 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acdded14-6f01-4f17-97ea-7b9f7f1006eb-config-volume\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:06.151527 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.151426 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:06.151527 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.151467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acdded14-6f01-4f17-97ea-7b9f7f1006eb-tmp-dir\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:06.252693 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.252591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.252693 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.252645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:06.252927 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.252698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acdded14-6f01-4f17-97ea-7b9f7f1006eb-tmp-dir\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:06.252927 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.252729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-878bf\" (UniqueName: \"kubernetes.io/projected/d09fba07-6330-44b8-b463-0a48b3d9e70b-kube-api-access-878bf\") pod \"ingress-canary-d8gdx\" (UID: \"d09fba07-6330-44b8-b463-0a48b3d9e70b\") " pod="openshift-ingress-canary/ingress-canary-d8gdx" Sep 08 10:02:06.252927 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:06.252731 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 10:02:06.252927 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.252761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d09fba07-6330-44b8-b463-0a48b3d9e70b-cert\") pod \"ingress-canary-d8gdx\" (UID: \"d09fba07-6330-44b8-b463-0a48b3d9e70b\") " pod="openshift-ingress-canary/ingress-canary-d8gdx" Sep 08 10:02:06.252927 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:06.252784 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls podName:acdded14-6f01-4f17-97ea-7b9f7f1006eb nodeName:}" failed. No retries permitted until 2026-09-08 10:02:06.752769841 +0000 UTC m=+36.541821623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls") pod "dns-default-n64nz" (UID: "acdded14-6f01-4f17-97ea-7b9f7f1006eb") : secret "dns-default-metrics-tls" not found Sep 08 10:02:06.252927 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.252881 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-data-volume\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.252927 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.252900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.252927 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.252924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t25t8\" (UniqueName: \"kubernetes.io/projected/acdded14-6f01-4f17-97ea-7b9f7f1006eb-kube-api-access-t25t8\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:06.253273 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.252945 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmcnf\" (UniqueName: \"kubernetes.io/projected/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-api-access-fmcnf\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.253273 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.252977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-crio-socket\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.253273 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.253031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acdded14-6f01-4f17-97ea-7b9f7f1006eb-config-volume\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:06.253273 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.253069 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acdded14-6f01-4f17-97ea-7b9f7f1006eb-tmp-dir\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:06.253499 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.253482 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acdded14-6f01-4f17-97ea-7b9f7f1006eb-config-volume\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:06.264299 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.264174 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t25t8\" (UniqueName: \"kubernetes.io/projected/acdded14-6f01-4f17-97ea-7b9f7f1006eb-kube-api-access-t25t8\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:06.353835 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.353793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-data-volume\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.353835 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.353833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.354041 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.353864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmcnf\" (UniqueName: \"kubernetes.io/projected/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-api-access-fmcnf\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.354041 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:06.353994 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 10:02:06.354126 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:06.354076 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls podName:fb5984d1-f23f-49fd-95a3-aaa11bb607a0 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:06.854055366 +0000 UTC m=+36.643107167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hqtdm" (UID: "fb5984d1-f23f-49fd-95a3-aaa11bb607a0") : secret "insights-runtime-extractor-tls" not found Sep 08 10:02:06.354126 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.354099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-crio-socket\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.354207 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.354161 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-data-volume\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.354207 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.354175 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.354286 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.354216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-878bf\" (UniqueName: \"kubernetes.io/projected/d09fba07-6330-44b8-b463-0a48b3d9e70b-kube-api-access-878bf\") pod \"ingress-canary-d8gdx\" (UID: \"d09fba07-6330-44b8-b463-0a48b3d9e70b\") " pod="openshift-ingress-canary/ingress-canary-d8gdx" Sep 08 10:02:06.354286 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.354247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d09fba07-6330-44b8-b463-0a48b3d9e70b-cert\") pod \"ingress-canary-d8gdx\" (UID: \"d09fba07-6330-44b8-b463-0a48b3d9e70b\") " pod="openshift-ingress-canary/ingress-canary-d8gdx" Sep 08 10:02:06.354358 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:06.354293 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 10:02:06.354389 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.354360 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-crio-socket\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.354428 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:06.354387 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-rbac-proxy-cm podName:fb5984d1-f23f-49fd-95a3-aaa11bb607a0 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:06.854371284 +0000 UTC m=+36.643423083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hqtdm" (UID: "fb5984d1-f23f-49fd-95a3-aaa11bb607a0") : configmap "kube-rbac-proxy" not found Sep 08 10:02:06.356688 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.356665 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d09fba07-6330-44b8-b463-0a48b3d9e70b-cert\") pod \"ingress-canary-d8gdx\" (UID: \"d09fba07-6330-44b8-b463-0a48b3d9e70b\") " pod="openshift-ingress-canary/ingress-canary-d8gdx" Sep 08 10:02:06.363218 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.363188 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmcnf\" (UniqueName: \"kubernetes.io/projected/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-api-access-fmcnf\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.363448 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.363429 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-878bf\" (UniqueName: \"kubernetes.io/projected/d09fba07-6330-44b8-b463-0a48b3d9e70b-kube-api-access-878bf\") pod \"ingress-canary-d8gdx\" (UID: \"d09fba07-6330-44b8-b463-0a48b3d9e70b\") " pod="openshift-ingress-canary/ingress-canary-d8gdx" Sep 08 10:02:06.410852 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.410821 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-9xf92"] Sep 08 10:02:06.421079 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.421051 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d8gdx" Sep 08 10:02:06.439388 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.439360 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-9xf92"] Sep 08 10:02:06.439543 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.439454 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-9xf92" Sep 08 10:02:06.441996 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.441972 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-sq72t\"" Sep 08 10:02:06.442223 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.442206 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 08 10:02:06.442482 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.442464 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 08 10:02:06.442561 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.442484 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 08 10:02:06.442561 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.442531 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 08 10:02:06.468072 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.468041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:06.526285 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.526252 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:06.556497 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.556456 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3-signing-cabundle\") pod \"service-ca-75d99c484-9xf92\" (UID: \"bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3\") " pod="openshift-service-ca/service-ca-75d99c484-9xf92" Sep 08 10:02:06.556704 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.556549 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97qw2\" (UniqueName: \"kubernetes.io/projected/bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3-kube-api-access-97qw2\") pod \"service-ca-75d99c484-9xf92\" (UID: \"bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3\") " pod="openshift-service-ca/service-ca-75d99c484-9xf92" Sep 08 10:02:06.556704 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.556584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3-signing-key\") pod \"service-ca-75d99c484-9xf92\" (UID: \"bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3\") " pod="openshift-service-ca/service-ca-75d99c484-9xf92" Sep 08 10:02:06.656938 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.656901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qw2\" (UniqueName: \"kubernetes.io/projected/bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3-kube-api-access-97qw2\") pod \"service-ca-75d99c484-9xf92\" (UID: \"bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3\") " pod="openshift-service-ca/service-ca-75d99c484-9xf92" Sep 08 10:02:06.657115 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.656945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3-signing-key\") pod \"service-ca-75d99c484-9xf92\" (UID: \"bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3\") " pod="openshift-service-ca/service-ca-75d99c484-9xf92" Sep 08 10:02:06.657179 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.657111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3-signing-cabundle\") pod \"service-ca-75d99c484-9xf92\" (UID: \"bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3\") " pod="openshift-service-ca/service-ca-75d99c484-9xf92" Sep 08 10:02:06.657767 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.657744 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3-signing-cabundle\") pod \"service-ca-75d99c484-9xf92\" (UID: \"bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3\") " pod="openshift-service-ca/service-ca-75d99c484-9xf92" Sep 08 10:02:06.659490 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.659470 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3-signing-key\") pod \"service-ca-75d99c484-9xf92\" (UID: \"bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3\") " pod="openshift-service-ca/service-ca-75d99c484-9xf92" Sep 08 10:02:06.667112 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.667093 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97qw2\" (UniqueName: \"kubernetes.io/projected/bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3-kube-api-access-97qw2\") pod \"service-ca-75d99c484-9xf92\" (UID: \"bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3\") " pod="openshift-service-ca/service-ca-75d99c484-9xf92" Sep 08 10:02:06.731862 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.731823 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d8gdx"] Sep 08 10:02:06.736179 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:02:06.736147 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09fba07_6330_44b8_b463_0a48b3d9e70b.slice/crio-0510fee5680a70446c765db2651b999671c31358a518c8eaeb6252b9c9fd594e WatchSource:0}: Error finding container 0510fee5680a70446c765db2651b999671c31358a518c8eaeb6252b9c9fd594e: Status 404 returned error can't find the container with id 0510fee5680a70446c765db2651b999671c31358a518c8eaeb6252b9c9fd594e Sep 08 10:02:06.748084 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.748061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-9xf92" Sep 08 10:02:06.758397 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.758373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:06.758544 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:06.758525 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 10:02:06.758591 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:06.758586 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls podName:acdded14-6f01-4f17-97ea-7b9f7f1006eb nodeName:}" failed. No retries permitted until 2026-09-08 10:02:07.758571581 +0000 UTC m=+37.547623363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls") pod "dns-default-n64nz" (UID: "acdded14-6f01-4f17-97ea-7b9f7f1006eb") : secret "dns-default-metrics-tls" not found Sep 08 10:02:06.830948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.830920 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-95fpk_d5114137-2635-4a57-bdae-7866a1b417a2/dns-node-resolver/0.log" Sep 08 10:02:06.852113 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.852082 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d8gdx" event={"ID":"d09fba07-6330-44b8-b463-0a48b3d9e70b","Type":"ContainerStarted","Data":"0510fee5680a70446c765db2651b999671c31358a518c8eaeb6252b9c9fd594e"} Sep 08 10:02:06.859408 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.859384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.859498 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:06.859477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:06.859555 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:06.859540 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 10:02:06.859590 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:06.859558 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 10:02:06.859656 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:06.859605 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls podName:fb5984d1-f23f-49fd-95a3-aaa11bb607a0 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:07.859590541 +0000 UTC m=+37.648642329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hqtdm" (UID: "fb5984d1-f23f-49fd-95a3-aaa11bb607a0") : secret "insights-runtime-extractor-tls" not found Sep 08 10:02:06.859656 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:06.859645 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-rbac-proxy-cm podName:fb5984d1-f23f-49fd-95a3-aaa11bb607a0 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:07.859634668 +0000 UTC m=+37.648686471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hqtdm" (UID: "fb5984d1-f23f-49fd-95a3-aaa11bb607a0") : configmap "kube-rbac-proxy" not found Sep 08 10:02:07.073994 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:07.073910 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-9xf92"] Sep 08 10:02:07.077790 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:02:07.077765 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc86e905_8ec1_494d_bd6e_fc9a1c1f22f3.slice/crio-9fec88bb1febd89cc6233a31e30d7987fdd4bfd2b51f035b14e9245d96793a0a WatchSource:0}: Error finding container 9fec88bb1febd89cc6233a31e30d7987fdd4bfd2b51f035b14e9245d96793a0a: Status 404 returned error can't find the container with id 9fec88bb1febd89cc6233a31e30d7987fdd4bfd2b51f035b14e9245d96793a0a Sep 08 10:02:07.671488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:07.671443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:02:07.671488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:07.671470 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:02:07.671757 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:07.671452 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:02:07.674049 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:07.674025 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 10:02:07.675001 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:07.674884 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 10:02:07.675001 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:07.674916 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8nq8l\"" Sep 08 10:02:07.675001 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:07.674917 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v9j5d\"" Sep 08 10:02:07.675001 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:07.674942 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 10:02:07.675295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:07.675226 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 10:02:07.766166 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:07.766136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:07.766367 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:07.766286 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 10:02:07.766367 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:07.766357 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls podName:acdded14-6f01-4f17-97ea-7b9f7f1006eb nodeName:}" failed. No retries permitted until 2026-09-08 10:02:09.766339012 +0000 UTC m=+39.555390808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls") pod "dns-default-n64nz" (UID: "acdded14-6f01-4f17-97ea-7b9f7f1006eb") : secret "dns-default-metrics-tls" not found Sep 08 10:02:07.831372 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:07.831343 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wqx5v_12807ade-0ca9-4bd3-b277-afba07e51a26/node-ca/0.log" Sep 08 10:02:07.856468 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:07.856430 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-9xf92" event={"ID":"bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3","Type":"ContainerStarted","Data":"9fec88bb1febd89cc6233a31e30d7987fdd4bfd2b51f035b14e9245d96793a0a"} Sep 08 10:02:07.866911 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:07.866875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:07.867101 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:07.867001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:07.867101 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:07.867032 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 10:02:07.867101 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:07.867095 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 10:02:07.867216 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:07.867108 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls podName:fb5984d1-f23f-49fd-95a3-aaa11bb607a0 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:09.867091215 +0000 UTC m=+39.656143014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hqtdm" (UID: "fb5984d1-f23f-49fd-95a3-aaa11bb607a0") : secret "insights-runtime-extractor-tls" not found Sep 08 10:02:07.867216 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:07.867153 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-rbac-proxy-cm podName:fb5984d1-f23f-49fd-95a3-aaa11bb607a0 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:09.867136785 +0000 UTC m=+39.656188585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hqtdm" (UID: "fb5984d1-f23f-49fd-95a3-aaa11bb607a0") : configmap "kube-rbac-proxy" not found Sep 08 10:02:08.174540 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:08.174501 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:08.286330 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:08.286295 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:08.803446 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:08.803406 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:09.782996 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:09.782958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:09.783475 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:09.783100 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 10:02:09.783475 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:09.783169 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls podName:acdded14-6f01-4f17-97ea-7b9f7f1006eb nodeName:}" failed. No retries permitted until 2026-09-08 10:02:13.783152408 +0000 UTC m=+43.572204203 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls") pod "dns-default-n64nz" (UID: "acdded14-6f01-4f17-97ea-7b9f7f1006eb") : secret "dns-default-metrics-tls" not found Sep 08 10:02:09.884066 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:09.884031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:09.884229 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:09.884165 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 10:02:09.884229 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:09.884201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:09.884339 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:09.884233 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-rbac-proxy-cm podName:fb5984d1-f23f-49fd-95a3-aaa11bb607a0 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:13.88421333 +0000 UTC m=+43.673265117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hqtdm" (UID: "fb5984d1-f23f-49fd-95a3-aaa11bb607a0") : configmap "kube-rbac-proxy" not found Sep 08 10:02:09.884339 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:09.884316 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 10:02:09.884454 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:09.884373 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls podName:fb5984d1-f23f-49fd-95a3-aaa11bb607a0 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:13.884357836 +0000 UTC m=+43.673409621 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hqtdm" (UID: "fb5984d1-f23f-49fd-95a3-aaa11bb607a0") : secret "insights-runtime-extractor-tls" not found Sep 08 10:02:10.321748 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:10.321714 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:10.867503 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:10.867454 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d8gdx" event={"ID":"d09fba07-6330-44b8-b463-0a48b3d9e70b","Type":"ContainerStarted","Data":"204d8f42241632dc6f61c0c099723a3e51032e110cf1f6df50fd974cd8521246"} Sep 08 10:02:10.884967 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:10.884909 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d8gdx" podStartSLOduration=1.89148995 podStartE2EDuration="4.884887996s" totalImagesPullingTime="2.993398046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:10.884065847 +0000 UTC m=+40.673117652" watchObservedRunningTime="2026-09-08 10:02:10.884887996 +0000 UTC m=+40.673939805" Sep 08 10:02:11.871562 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:11.871470 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-9xf92" event={"ID":"bc86e905-8ec1-494d-bd6e-fc9a1c1f22f3","Type":"ContainerStarted","Data":"a667feaa024a110db6a11705b619edfb5dc55081cd6209c14dce4401c3110140"} Sep 08 10:02:11.886660 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:11.886593 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-9xf92" podStartSLOduration=1.5366792999999999 podStartE2EDuration="5.886576511s" totalImagesPullingTime="4.349897211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:11.88510528 +0000 UTC m=+41.674157084" watchObservedRunningTime="2026-09-08 10:02:11.886576511 +0000 UTC m=+41.675628354" Sep 08 10:02:12.513861 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:12.513828 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:12.748837 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:12.748792 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:12.896968 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:12.896937 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:13.058657 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:13.058607 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:13.704567 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:13.704531 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:13.805474 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:13.805446 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:13.824553 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:13.824525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:13.824673 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:13.824649 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 10:02:13.824714 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:13.824701 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls podName:acdded14-6f01-4f17-97ea-7b9f7f1006eb nodeName:}" failed. No retries permitted until 2026-09-08 10:02:21.824687736 +0000 UTC m=+51.613739518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls") pod "dns-default-n64nz" (UID: "acdded14-6f01-4f17-97ea-7b9f7f1006eb") : secret "dns-default-metrics-tls" not found Sep 08 10:02:13.925349 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:13.925319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:13.925737 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:13.925382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:13.925737 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:13.925500 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 10:02:13.925737 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:13.925559 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls podName:fb5984d1-f23f-49fd-95a3-aaa11bb607a0 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:21.925545786 +0000 UTC m=+51.714597568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hqtdm" (UID: "fb5984d1-f23f-49fd-95a3-aaa11bb607a0") : secret "insights-runtime-extractor-tls" not found Sep 08 10:02:13.925900 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:13.925880 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:14.624634 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:14.624583 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:14.990932 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:14.990894 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:15.110682 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:15.110649 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:15.211241 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:15.211208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:15.389388 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:15.389312 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:15.513352 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:15.513318 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:15.801070 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:15.801037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:16.190593 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:16.190560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:16.300287 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:16.300256 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:16.596998 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:16.596916 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:16.995964 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:16.995927 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:19.194735 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:19.194613 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:19.267256 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:19.267230 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:19.395678 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:19.395643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:19.733742 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:19.733710 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:21.250429 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:21.250388 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:21.882684 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:21.882636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:21.886002 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:21.885977 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acdded14-6f01-4f17-97ea-7b9f7f1006eb-metrics-tls\") pod \"dns-default-n64nz\" (UID: \"acdded14-6f01-4f17-97ea-7b9f7f1006eb\") " pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:21.948071 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:21.948038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:21.983478 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:21.983441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:21.983994 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:21.983964 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:21.985923 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:21.985893 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5984d1-f23f-49fd-95a3-aaa11bb607a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hqtdm\" (UID: \"fb5984d1-f23f-49fd-95a3-aaa11bb607a0\") " pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:22.000625 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:22.000597 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hqtdm" Sep 08 10:02:22.051032 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:22.050745 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:22.190957 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:22.190922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:22.268924 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:22.268889 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n64nz"] Sep 08 10:02:22.271949 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:02:22.271927 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacdded14_6f01_4f17_97ea_7b9f7f1006eb.slice/crio-1c3beadb678d7aa70e1987c3a9c580f28df1a01459e57173bb932cd06f316caf WatchSource:0}: Error finding container 1c3beadb678d7aa70e1987c3a9c580f28df1a01459e57173bb932cd06f316caf: Status 404 returned error can't find the container with id 1c3beadb678d7aa70e1987c3a9c580f28df1a01459e57173bb932cd06f316caf Sep 08 10:02:22.291738 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:22.291711 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hqtdm"] Sep 08 10:02:22.294330 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:02:22.294307 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5984d1_f23f_49fd_95a3_aaa11bb607a0.slice/crio-1e5a89c745f45c7298f4fad5b7f09e6ead92703cbee12335777207dfa91b611a WatchSource:0}: Error finding container 1e5a89c745f45c7298f4fad5b7f09e6ead92703cbee12335777207dfa91b611a: Status 404 returned error can't find the container with id 1e5a89c745f45c7298f4fad5b7f09e6ead92703cbee12335777207dfa91b611a Sep 08 10:02:22.388030 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:22.387995 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:22.900720 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:22.900681 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hqtdm" event={"ID":"fb5984d1-f23f-49fd-95a3-aaa11bb607a0","Type":"ContainerStarted","Data":"807f6c4f34a18e445b5fb3e0a66c9a57c661274718af3173877a7fe5fc099548"} Sep 08 10:02:22.900876 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:22.900742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hqtdm" event={"ID":"fb5984d1-f23f-49fd-95a3-aaa11bb607a0","Type":"ContainerStarted","Data":"1e5a89c745f45c7298f4fad5b7f09e6ead92703cbee12335777207dfa91b611a"} Sep 08 10:02:22.901710 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:22.901689 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n64nz" event={"ID":"acdded14-6f01-4f17-97ea-7b9f7f1006eb","Type":"ContainerStarted","Data":"1c3beadb678d7aa70e1987c3a9c580f28df1a01459e57173bb932cd06f316caf"} Sep 08 10:02:22.999077 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:22.999039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:23.193060 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:23.192964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:25.292141 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:25.292086 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:25.393230 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:25.393186 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:25.494137 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:25.494096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:26.915030 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:26.914992 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hqtdm" event={"ID":"fb5984d1-f23f-49fd-95a3-aaa11bb607a0","Type":"ContainerStarted","Data":"44c866e037fb2fdc421ae7005b4cd1b131bb67432058c055305c4986c38aa899"} Sep 08 10:02:26.916964 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:26.916926 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n64nz" event={"ID":"acdded14-6f01-4f17-97ea-7b9f7f1006eb","Type":"ContainerStarted","Data":"eaf855a2330d7f0573e658c7494cafa4b6e51e7ed288042a97618ecc6ffb0dd2"} Sep 08 10:02:26.917092 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:26.916978 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n64nz" event={"ID":"acdded14-6f01-4f17-97ea-7b9f7f1006eb","Type":"ContainerStarted","Data":"bd0e3f4cbd9c09edca335bd5967914aba489863b0679ffef0881c0ff2e2d71a8"} Sep 08 10:02:26.917092 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:26.917030 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:26.933065 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:26.933024 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n64nz" podStartSLOduration=17.069761373 podStartE2EDuration="20.933009401s" totalImagesPullingTime="3.863248028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:26.932333556 +0000 UTC m=+56.721385361" watchObservedRunningTime="2026-09-08 10:02:26.933009401 +0000 UTC m=+56.722061196" Sep 08 10:02:27.357504 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:27.357422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:27.457110 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:27.457076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:27.922089 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:27.922059 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hqtdm" event={"ID":"fb5984d1-f23f-49fd-95a3-aaa11bb607a0","Type":"ContainerStarted","Data":"78d462c8f16233d2d01c6ac5136b3d83118ed8f421d04468af228e7cd6404499"} Sep 08 10:02:27.941434 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:27.941386 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hqtdm" podStartSLOduration=16.766844533 podStartE2EDuration="21.941372535s" totalImagesPullingTime="5.174528002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:02:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:27.941124291 +0000 UTC m=+57.730176094" watchObservedRunningTime="2026-09-08 10:02:27.941372535 +0000 UTC m=+57.730424338" Sep 08 10:02:28.212983 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:28.212889 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:28.311824 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:28.311791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:29.295633 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.295585 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d656c685b-zqmc2"] Sep 08 10:02:29.297805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.297788 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d656c685b-zqmc2" Sep 08 10:02:29.301943 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.301914 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 08 10:02:29.302760 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.302740 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 08 10:02:29.303353 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.303334 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 08 10:02:29.304868 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.304849 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-6d9w8\"" Sep 08 10:02:29.305061 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.305042 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 08 10:02:29.306760 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.306741 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb"] Sep 08 10:02:29.308509 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.308488 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" Sep 08 10:02:29.311005 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.310978 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d656c685b-zqmc2"] Sep 08 10:02:29.311165 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.311144 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 08 10:02:29.341981 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.341947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06154cd0-0bba-40db-ab87-629fee28674f-tmp\") pod \"klusterlet-addon-workmgr-5f87485948-zgclb\" (UID: \"06154cd0-0bba-40db-ab87-629fee28674f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" Sep 08 10:02:29.342171 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.342038 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/06154cd0-0bba-40db-ab87-629fee28674f-klusterlet-config\") pod \"klusterlet-addon-workmgr-5f87485948-zgclb\" (UID: \"06154cd0-0bba-40db-ab87-629fee28674f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" Sep 08 10:02:29.342171 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.342094 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b6bcb06-f0ea-46d0-874f-1de4ec47aa80-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d656c685b-zqmc2\" (UID: \"6b6bcb06-f0ea-46d0-874f-1de4ec47aa80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d656c685b-zqmc2" Sep 08 10:02:29.342171 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.342122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz5x2\" (UniqueName: \"kubernetes.io/projected/6b6bcb06-f0ea-46d0-874f-1de4ec47aa80-kube-api-access-pz5x2\") pod \"managed-serviceaccount-addon-agent-d656c685b-zqmc2\" (UID: \"6b6bcb06-f0ea-46d0-874f-1de4ec47aa80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d656c685b-zqmc2" Sep 08 10:02:29.342171 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.342154 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87s2c\" (UniqueName: \"kubernetes.io/projected/06154cd0-0bba-40db-ab87-629fee28674f-kube-api-access-87s2c\") pod \"klusterlet-addon-workmgr-5f87485948-zgclb\" (UID: \"06154cd0-0bba-40db-ab87-629fee28674f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" Sep 08 10:02:29.345767 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.345737 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb"] Sep 08 10:02:29.399908 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.399876 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-qhcfl"] Sep 08 10:02:29.401802 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.401779 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-qhcfl" Sep 08 10:02:29.404288 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.404262 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-t72lv\"" Sep 08 10:02:29.404596 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.404578 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 08 10:02:29.404695 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.404667 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 08 10:02:29.405294 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.405276 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6666db957f-c2qnq"] Sep 08 10:02:29.407130 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.407111 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.410397 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.410376 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9wz2f\"" Sep 08 10:02:29.410532 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.410407 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 08 10:02:29.410580 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.410555 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 08 10:02:29.410686 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.410651 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 08 10:02:29.415510 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.415493 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 08 10:02:29.423997 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.423797 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-qhcfl"] Sep 08 10:02:29.424688 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.424670 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:29.424745 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.424694 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6666db957f-c2qnq"] Sep 08 10:02:29.442508 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.442482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f4222e2-59da-4454-bc94-a6fbee924a54-image-registry-private-configuration\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.442637 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.442519 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/06154cd0-0bba-40db-ab87-629fee28674f-klusterlet-config\") pod \"klusterlet-addon-workmgr-5f87485948-zgclb\" (UID: \"06154cd0-0bba-40db-ab87-629fee28674f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" Sep 08 10:02:29.442637 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.442586 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f4222e2-59da-4454-bc94-a6fbee924a54-installation-pull-secrets\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.442713 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.442638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f4222e2-59da-4454-bc94-a6fbee924a54-bound-sa-token\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.442713 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.442687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b6bcb06-f0ea-46d0-874f-1de4ec47aa80-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d656c685b-zqmc2\" (UID: \"6b6bcb06-f0ea-46d0-874f-1de4ec47aa80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d656c685b-zqmc2" Sep 08 10:02:29.442778 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.442716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f4222e2-59da-4454-bc94-a6fbee924a54-registry-certificates\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.442778 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.442747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz5x2\" (UniqueName: \"kubernetes.io/projected/6b6bcb06-f0ea-46d0-874f-1de4ec47aa80-kube-api-access-pz5x2\") pod \"managed-serviceaccount-addon-agent-d656c685b-zqmc2\" (UID: \"6b6bcb06-f0ea-46d0-874f-1de4ec47aa80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d656c685b-zqmc2" Sep 08 10:02:29.442869 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.442776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87s2c\" (UniqueName: \"kubernetes.io/projected/06154cd0-0bba-40db-ab87-629fee28674f-kube-api-access-87s2c\") pod \"klusterlet-addon-workmgr-5f87485948-zgclb\" (UID: \"06154cd0-0bba-40db-ab87-629fee28674f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" Sep 08 10:02:29.442869 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.442806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06154cd0-0bba-40db-ab87-629fee28674f-tmp\") pod \"klusterlet-addon-workmgr-5f87485948-zgclb\" (UID: \"06154cd0-0bba-40db-ab87-629fee28674f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" Sep 08 10:02:29.442869 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.442833 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f4222e2-59da-4454-bc94-a6fbee924a54-registry-tls\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.442869 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.442859 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f4222e2-59da-4454-bc94-a6fbee924a54-ca-trust-extracted\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.443051 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.442900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f4222e2-59da-4454-bc94-a6fbee924a54-trusted-ca\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.443051 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.442928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5hkr\" (UniqueName: \"kubernetes.io/projected/9f4222e2-59da-4454-bc94-a6fbee924a54-kube-api-access-g5hkr\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.443051 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.442960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zts9j\" (UniqueName: \"kubernetes.io/projected/00a89ef4-4cc0-4f8c-b01f-b8dfa24e340a-kube-api-access-zts9j\") pod \"downloads-85d98c998f-qhcfl\" (UID: \"00a89ef4-4cc0-4f8c-b01f-b8dfa24e340a\") " pod="openshift-console/downloads-85d98c998f-qhcfl" Sep 08 10:02:29.443292 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.443271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06154cd0-0bba-40db-ab87-629fee28674f-tmp\") pod \"klusterlet-addon-workmgr-5f87485948-zgclb\" (UID: \"06154cd0-0bba-40db-ab87-629fee28674f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" Sep 08 10:02:29.445593 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.445572 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/06154cd0-0bba-40db-ab87-629fee28674f-klusterlet-config\") pod \"klusterlet-addon-workmgr-5f87485948-zgclb\" (UID: \"06154cd0-0bba-40db-ab87-629fee28674f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" Sep 08 10:02:29.445711 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.445596 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b6bcb06-f0ea-46d0-874f-1de4ec47aa80-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d656c685b-zqmc2\" (UID: \"6b6bcb06-f0ea-46d0-874f-1de4ec47aa80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d656c685b-zqmc2" Sep 08 10:02:29.468351 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.468328 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz5x2\" (UniqueName: \"kubernetes.io/projected/6b6bcb06-f0ea-46d0-874f-1de4ec47aa80-kube-api-access-pz5x2\") pod \"managed-serviceaccount-addon-agent-d656c685b-zqmc2\" (UID: \"6b6bcb06-f0ea-46d0-874f-1de4ec47aa80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d656c685b-zqmc2" Sep 08 10:02:29.468581 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.468564 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87s2c\" (UniqueName: \"kubernetes.io/projected/06154cd0-0bba-40db-ab87-629fee28674f-kube-api-access-87s2c\") pod \"klusterlet-addon-workmgr-5f87485948-zgclb\" (UID: \"06154cd0-0bba-40db-ab87-629fee28674f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" Sep 08 10:02:29.507716 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.507685 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:29.544060 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.544040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f4222e2-59da-4454-bc94-a6fbee924a54-installation-pull-secrets\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.544209 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.544068 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f4222e2-59da-4454-bc94-a6fbee924a54-bound-sa-token\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.544209 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.544091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f4222e2-59da-4454-bc94-a6fbee924a54-registry-certificates\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.544209 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.544116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f4222e2-59da-4454-bc94-a6fbee924a54-registry-tls\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.544349 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.544245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f4222e2-59da-4454-bc94-a6fbee924a54-ca-trust-extracted\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.544349 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.544297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f4222e2-59da-4454-bc94-a6fbee924a54-trusted-ca\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.544349 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.544328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5hkr\" (UniqueName: \"kubernetes.io/projected/9f4222e2-59da-4454-bc94-a6fbee924a54-kube-api-access-g5hkr\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.544494 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.544356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zts9j\" (UniqueName: \"kubernetes.io/projected/00a89ef4-4cc0-4f8c-b01f-b8dfa24e340a-kube-api-access-zts9j\") pod \"downloads-85d98c998f-qhcfl\" (UID: \"00a89ef4-4cc0-4f8c-b01f-b8dfa24e340a\") " pod="openshift-console/downloads-85d98c998f-qhcfl" Sep 08 10:02:29.544494 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.544420 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f4222e2-59da-4454-bc94-a6fbee924a54-image-registry-private-configuration\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.545046 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.545019 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f4222e2-59da-4454-bc94-a6fbee924a54-ca-trust-extracted\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.545161 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.545094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f4222e2-59da-4454-bc94-a6fbee924a54-registry-certificates\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.545841 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.545796 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f4222e2-59da-4454-bc94-a6fbee924a54-trusted-ca\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.546666 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.546642 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f4222e2-59da-4454-bc94-a6fbee924a54-installation-pull-secrets\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.546763 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.546710 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f4222e2-59da-4454-bc94-a6fbee924a54-registry-tls\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.546807 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.546779 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f4222e2-59da-4454-bc94-a6fbee924a54-image-registry-private-configuration\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.552172 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.552145 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5hkr\" (UniqueName: \"kubernetes.io/projected/9f4222e2-59da-4454-bc94-a6fbee924a54-kube-api-access-g5hkr\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.552353 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.552337 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f4222e2-59da-4454-bc94-a6fbee924a54-bound-sa-token\") pod \"image-registry-6666db957f-c2qnq\" (UID: \"9f4222e2-59da-4454-bc94-a6fbee924a54\") " pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.552547 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.552529 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zts9j\" (UniqueName: \"kubernetes.io/projected/00a89ef4-4cc0-4f8c-b01f-b8dfa24e340a-kube-api-access-zts9j\") pod \"downloads-85d98c998f-qhcfl\" (UID: \"00a89ef4-4cc0-4f8c-b01f-b8dfa24e340a\") " pod="openshift-console/downloads-85d98c998f-qhcfl" Sep 08 10:02:29.607907 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.607877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:29.618872 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.618847 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d656c685b-zqmc2" Sep 08 10:02:29.626428 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.626411 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" Sep 08 10:02:29.708407 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.708379 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:29.711479 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.711461 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-qhcfl" Sep 08 10:02:29.717411 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.717382 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:29.885690 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.885598 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d656c685b-zqmc2"] Sep 08 10:02:29.889546 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:02:29.889524 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b6bcb06_f0ea_46d0_874f_1de4ec47aa80.slice/crio-4f27d0dabff1b8fe4daf81e6db59ff7be2cf4cfa309fd87ce29ea670901dd0c0 WatchSource:0}: Error finding container 4f27d0dabff1b8fe4daf81e6db59ff7be2cf4cfa309fd87ce29ea670901dd0c0: Status 404 returned error can't find the container with id 4f27d0dabff1b8fe4daf81e6db59ff7be2cf4cfa309fd87ce29ea670901dd0c0 Sep 08 10:02:29.897959 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.897936 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb"] Sep 08 10:02:29.900755 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:02:29.900732 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06154cd0_0bba_40db_ab87_629fee28674f.slice/crio-0a7b931004eea552052d0c3057096a175a23a5f96e334ee484094ce4c403e48d WatchSource:0}: Error finding container 0a7b931004eea552052d0c3057096a175a23a5f96e334ee484094ce4c403e48d: Status 404 returned error can't find the container with id 0a7b931004eea552052d0c3057096a175a23a5f96e334ee484094ce4c403e48d Sep 08 10:02:29.928942 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.928909 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" event={"ID":"06154cd0-0bba-40db-ab87-629fee28674f","Type":"ContainerStarted","Data":"0a7b931004eea552052d0c3057096a175a23a5f96e334ee484094ce4c403e48d"} Sep 08 10:02:29.929911 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.929890 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d656c685b-zqmc2" event={"ID":"6b6bcb06-f0ea-46d0-874f-1de4ec47aa80","Type":"ContainerStarted","Data":"4f27d0dabff1b8fe4daf81e6db59ff7be2cf4cfa309fd87ce29ea670901dd0c0"} Sep 08 10:02:29.990714 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.990611 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-qhcfl"] Sep 08 10:02:29.991358 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:29.991337 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6666db957f-c2qnq"] Sep 08 10:02:29.993298 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:02:29.993262 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a89ef4_4cc0_4f8c_b01f_b8dfa24e340a.slice/crio-004cdbafca337cd20552a8fba6870e1c8ad5ee963e2cf47b7d617e4da53039e1 WatchSource:0}: Error finding container 004cdbafca337cd20552a8fba6870e1c8ad5ee963e2cf47b7d617e4da53039e1: Status 404 returned error can't find the container with id 004cdbafca337cd20552a8fba6870e1c8ad5ee963e2cf47b7d617e4da53039e1 Sep 08 10:02:29.993860 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:02:29.993844 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4222e2_59da_4454_bc94_a6fbee924a54.slice/crio-a4d414d2c49f34bbd0145906b5c5b585fb33d1e1796f88be54ec3bef001b6431 WatchSource:0}: Error finding container a4d414d2c49f34bbd0145906b5c5b585fb33d1e1796f88be54ec3bef001b6431: Status 404 returned error can't find the container with id a4d414d2c49f34bbd0145906b5c5b585fb33d1e1796f88be54ec3bef001b6431 Sep 08 10:02:30.024564 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.024540 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f7448cf6-v87wv"] Sep 08 10:02:30.027191 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.027170 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.029356 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.029315 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-brk64\"" Sep 08 10:02:30.029448 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.029376 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 10:02:30.029448 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.029435 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 10:02:30.030060 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.030046 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 10:02:30.030135 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.030093 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 10:02:30.030181 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.030132 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 10:02:30.038171 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.038150 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f7448cf6-v87wv"] Sep 08 10:02:30.048418 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.048396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-config\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.048511 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.048432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-oauth-serving-cert\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.048558 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.048506 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-oauth-config\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.048558 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.048536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-serving-cert\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.048714 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.048596 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-service-ca\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.048714 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.048629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pddpc\" (UniqueName: \"kubernetes.io/projected/a88ac405-fc3e-47c7-bb68-313c148de5ef-kube-api-access-pddpc\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.149746 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.149697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-config\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.149746 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.149749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-oauth-serving-cert\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.149944 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.149895 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-oauth-config\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.149997 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.149949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-serving-cert\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.150075 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.150059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-service-ca\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.150128 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.150091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pddpc\" (UniqueName: \"kubernetes.io/projected/a88ac405-fc3e-47c7-bb68-313c148de5ef-kube-api-access-pddpc\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.150487 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.150460 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-config\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.150600 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.150537 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-oauth-serving-cert\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.150682 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.150668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-service-ca\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.152375 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.152348 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-serving-cert\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.152445 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.152383 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-oauth-config\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.157287 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.157267 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pddpc\" (UniqueName: \"kubernetes.io/projected/a88ac405-fc3e-47c7-bb68-313c148de5ef-kube-api-access-pddpc\") pod \"console-6f7448cf6-v87wv\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.335948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.335908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:30.434054 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.433979 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:30.605686 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.605650 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f7448cf6-v87wv"] Sep 08 10:02:30.608980 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:02:30.608956 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88ac405_fc3e_47c7_bb68_313c148de5ef.slice/crio-e9c82a8b9753d1238513b3e2f40af8eb9fd9fda35feb28de6956029a2bc3eb18 WatchSource:0}: Error finding container e9c82a8b9753d1238513b3e2f40af8eb9fd9fda35feb28de6956029a2bc3eb18: Status 404 returned error can't find the container with id e9c82a8b9753d1238513b3e2f40af8eb9fd9fda35feb28de6956029a2bc3eb18 Sep 08 10:02:30.937695 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.937655 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6666db957f-c2qnq" event={"ID":"9f4222e2-59da-4454-bc94-a6fbee924a54","Type":"ContainerStarted","Data":"83b1602f47d47f5c32729da12f49d2042c532b4775372e138959acd5e9697070"} Sep 08 10:02:30.937883 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.937733 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6666db957f-c2qnq" event={"ID":"9f4222e2-59da-4454-bc94-a6fbee924a54","Type":"ContainerStarted","Data":"a4d414d2c49f34bbd0145906b5c5b585fb33d1e1796f88be54ec3bef001b6431"} Sep 08 10:02:30.938383 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.938360 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:30.940543 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.940517 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-qhcfl" event={"ID":"00a89ef4-4cc0-4f8c-b01f-b8dfa24e340a","Type":"ContainerStarted","Data":"004cdbafca337cd20552a8fba6870e1c8ad5ee963e2cf47b7d617e4da53039e1"} Sep 08 10:02:30.942976 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.942914 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7448cf6-v87wv" event={"ID":"a88ac405-fc3e-47c7-bb68-313c148de5ef","Type":"ContainerStarted","Data":"e9c82a8b9753d1238513b3e2f40af8eb9fd9fda35feb28de6956029a2bc3eb18"} Sep 08 10:02:30.958429 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:30.957587 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6666db957f-c2qnq" podStartSLOduration=1.957567931 podStartE2EDuration="1.957567931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:30.956830188 +0000 UTC m=+60.745882008" watchObservedRunningTime="2026-09-08 10:02:30.957567931 +0000 UTC m=+60.746619752" Sep 08 10:02:31.525208 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:31.525144 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:31.602137 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:31.602101 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:31.665826 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:31.665743 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:31.854309 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:31.854238 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fcgv" Sep 08 10:02:31.954229 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:31.954186 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:32.060943 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:32.060877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:32.155503 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:32.155463 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:32.319035 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:32.319002 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:32.420874 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:32.420794 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:33.022960 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:33.022919 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:33.957063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:33.957022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:34.021126 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:34.021083 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:34.211341 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:34.211247 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:34.313209 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:34.313115 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:35.109842 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:35.109810 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:35.220968 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:35.220937 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:35.908593 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:35.908557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:36.415815 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.415734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs\") pod \"network-metrics-daemon-jgstq\" (UID: \"01480054-d16a-4820-af73-7a493b53ead0\") " pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:02:36.418469 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.418441 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 10:02:36.428613 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.428589 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01480054-d16a-4820-af73-7a493b53ead0-metrics-certs\") pod \"network-metrics-daemon-jgstq\" (UID: \"01480054-d16a-4820-af73-7a493b53ead0\") " pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:02:36.487254 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.487223 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8nq8l\"" Sep 08 10:02:36.495172 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.495137 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgstq" Sep 08 10:02:36.517236 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.517196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:02:36.519683 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.519655 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 10:02:36.529885 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.529857 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7ee997-0723-4a47-90b5-805cd7721382-original-pull-secret\") pod \"global-pull-secret-syncer-ktdpl\" (UID: \"2b7ee997-0723-4a47-90b5-805cd7721382\") " pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:02:36.618273 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.618238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9nw\" (UniqueName: \"kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw\") pod \"network-check-target-mvf7l\" (UID: \"334662a7-e44f-4aa9-8304-0a13dfed33e7\") " pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:02:36.622549 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.622520 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 10:02:36.631072 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.631039 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 10:02:36.642246 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.642215 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9nw\" (UniqueName: \"kubernetes.io/projected/334662a7-e44f-4aa9-8304-0a13dfed33e7-kube-api-access-cq9nw\") pod \"network-check-target-mvf7l\" (UID: \"334662a7-e44f-4aa9-8304-0a13dfed33e7\") " pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:02:36.701595 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.701512 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-d72f2"] Sep 08 10:02:36.706096 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.706074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" Sep 08 10:02:36.710720 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.710688 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 08 10:02:36.710852 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.710719 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 08 10:02:36.712190 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.711543 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 08 10:02:36.712190 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.711859 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6qpc6\"" Sep 08 10:02:36.712190 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.712071 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 08 10:02:36.712443 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.712294 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 08 10:02:36.724842 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.724818 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-d72f2"] Sep 08 10:02:36.776217 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.776186 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jgstq"] Sep 08 10:02:36.779343 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:02:36.779314 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01480054_d16a_4820_af73_7a493b53ead0.slice/crio-fa8903928ce5963b128aef05fc5b0e3f6db28e690cac75988a78b6967d086b25 WatchSource:0}: Error finding container fa8903928ce5963b128aef05fc5b0e3f6db28e690cac75988a78b6967d086b25: Status 404 returned error can't find the container with id fa8903928ce5963b128aef05fc5b0e3f6db28e690cac75988a78b6967d086b25 Sep 08 10:02:36.791294 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.791271 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktdpl" Sep 08 10:02:36.800307 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.800280 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v9j5d\"" Sep 08 10:02:36.808357 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.808323 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:02:36.819953 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.819917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01943a4b-bbca-418d-add3-d0169cd20bff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-d72f2\" (UID: \"01943a4b-bbca-418d-add3-d0169cd20bff\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" Sep 08 10:02:36.820180 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.820148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5q9\" (UniqueName: \"kubernetes.io/projected/01943a4b-bbca-418d-add3-d0169cd20bff-kube-api-access-bx5q9\") pod \"prometheus-operator-6f4946db56-d72f2\" (UID: \"01943a4b-bbca-418d-add3-d0169cd20bff\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" Sep 08 10:02:36.820276 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.820223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/01943a4b-bbca-418d-add3-d0169cd20bff-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-d72f2\" (UID: \"01943a4b-bbca-418d-add3-d0169cd20bff\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" Sep 08 10:02:36.820340 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.820297 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01943a4b-bbca-418d-add3-d0169cd20bff-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-d72f2\" (UID: \"01943a4b-bbca-418d-add3-d0169cd20bff\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" Sep 08 10:02:36.921473 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.920939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01943a4b-bbca-418d-add3-d0169cd20bff-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-d72f2\" (UID: \"01943a4b-bbca-418d-add3-d0169cd20bff\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" Sep 08 10:02:36.921473 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.921020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01943a4b-bbca-418d-add3-d0169cd20bff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-d72f2\" (UID: \"01943a4b-bbca-418d-add3-d0169cd20bff\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" Sep 08 10:02:36.921473 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.921057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5q9\" (UniqueName: \"kubernetes.io/projected/01943a4b-bbca-418d-add3-d0169cd20bff-kube-api-access-bx5q9\") pod \"prometheus-operator-6f4946db56-d72f2\" (UID: \"01943a4b-bbca-418d-add3-d0169cd20bff\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" Sep 08 10:02:36.921473 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.921083 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/01943a4b-bbca-418d-add3-d0169cd20bff-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-d72f2\" (UID: \"01943a4b-bbca-418d-add3-d0169cd20bff\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" Sep 08 10:02:36.921473 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:36.921227 2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 08 10:02:36.921473 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:36.921296 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01943a4b-bbca-418d-add3-d0169cd20bff-prometheus-operator-tls podName:01943a4b-bbca-418d-add3-d0169cd20bff nodeName:}" failed. No retries permitted until 2026-09-08 10:02:37.42127489 +0000 UTC m=+67.210326691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/01943a4b-bbca-418d-add3-d0169cd20bff-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-d72f2" (UID: "01943a4b-bbca-418d-add3-d0169cd20bff") : secret "prometheus-operator-tls" not found Sep 08 10:02:36.922144 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.922124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01943a4b-bbca-418d-add3-d0169cd20bff-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-d72f2\" (UID: \"01943a4b-bbca-418d-add3-d0169cd20bff\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" Sep 08 10:02:36.923966 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.923776 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01943a4b-bbca-418d-add3-d0169cd20bff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-d72f2\" (UID: \"01943a4b-bbca-418d-add3-d0169cd20bff\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" Sep 08 10:02:36.923966 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.923807 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n64nz" Sep 08 10:02:36.933423 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.933393 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5q9\" (UniqueName: \"kubernetes.io/projected/01943a4b-bbca-418d-add3-d0169cd20bff-kube-api-access-bx5q9\") pod \"prometheus-operator-6f4946db56-d72f2\" (UID: \"01943a4b-bbca-418d-add3-d0169cd20bff\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" Sep 08 10:02:36.970434 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.970158 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d656c685b-zqmc2" event={"ID":"6b6bcb06-f0ea-46d0-874f-1de4ec47aa80","Type":"ContainerStarted","Data":"15f21c5a9040aac95f0d362ba9e60737b1fda52b11eb3bc48b45e93a57f13f83"} Sep 08 10:02:36.972494 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.972409 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7448cf6-v87wv" event={"ID":"a88ac405-fc3e-47c7-bb68-313c148de5ef","Type":"ContainerStarted","Data":"ada2d3f20bffc1cbe509790d1ad2eb5a6d917670c1255f48c701d601b71a1091"} Sep 08 10:02:36.975472 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.975446 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" event={"ID":"06154cd0-0bba-40db-ab87-629fee28674f","Type":"ContainerStarted","Data":"420104e305db9aaacd12cca00bb1a540a4842d24d025acd8e806ff50abcb0ecb"} Sep 08 10:02:36.975659 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.975640 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" Sep 08 10:02:36.977414 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.977386 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jgstq" event={"ID":"01480054-d16a-4820-af73-7a493b53ead0","Type":"ContainerStarted","Data":"fa8903928ce5963b128aef05fc5b0e3f6db28e690cac75988a78b6967d086b25"} Sep 08 10:02:36.977605 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:36.977585 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" Sep 08 10:02:37.002961 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:37.002878 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d656c685b-zqmc2" podStartSLOduration=1.728389357 podStartE2EDuration="8.002860763s" totalImagesPullingTime="6.274471406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:36.984502655 +0000 UTC m=+66.773554460" watchObservedRunningTime="2026-09-08 10:02:37.002860763 +0000 UTC m=+66.791912569" Sep 08 10:02:37.003277 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:37.003244 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f87485948-zgclb" podStartSLOduration=1.7391196199999999 podStartE2EDuration="8.003236223s" totalImagesPullingTime="6.264116603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:37.002647871 +0000 UTC m=+66.791699672" watchObservedRunningTime="2026-09-08 10:02:37.003236223 +0000 UTC m=+66.792288027" Sep 08 10:02:37.028878 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:37.028827 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f7448cf6-v87wv" podStartSLOduration=1.462141824 podStartE2EDuration="7.028811381s" totalImagesPullingTime="5.566669557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:37.028451342 +0000 UTC m=+66.817503171" watchObservedRunningTime="2026-09-08 10:02:37.028811381 +0000 UTC m=+66.817863194" Sep 08 10:02:37.075725 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:37.075692 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ktdpl"] Sep 08 10:02:37.079114 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:02:37.079081 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7ee997_0723_4a47_90b5_805cd7721382.slice/crio-5a8d7fedaa5616dbf343638f4ab086793f4eedc9a67d51ea065ed3912a05d4a9 WatchSource:0}: Error finding container 5a8d7fedaa5616dbf343638f4ab086793f4eedc9a67d51ea065ed3912a05d4a9: Status 404 returned error can't find the container with id 5a8d7fedaa5616dbf343638f4ab086793f4eedc9a67d51ea065ed3912a05d4a9 Sep 08 10:02:37.122798 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:37.122767 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mvf7l"] Sep 08 10:02:37.126050 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:02:37.126020 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334662a7_e44f_4aa9_8304_0a13dfed33e7.slice/crio-622e0afbee9f665e5f1847704fc3be7f52946c2860bb50ce85f084ab9e40abaf WatchSource:0}: Error finding container 622e0afbee9f665e5f1847704fc3be7f52946c2860bb50ce85f084ab9e40abaf: Status 404 returned error can't find the container with id 622e0afbee9f665e5f1847704fc3be7f52946c2860bb50ce85f084ab9e40abaf Sep 08 10:02:37.425353 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:37.425318 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/01943a4b-bbca-418d-add3-d0169cd20bff-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-d72f2\" (UID: \"01943a4b-bbca-418d-add3-d0169cd20bff\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" Sep 08 10:02:37.427893 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:37.427864 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/01943a4b-bbca-418d-add3-d0169cd20bff-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-d72f2\" (UID: \"01943a4b-bbca-418d-add3-d0169cd20bff\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" Sep 08 10:02:37.617259 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:37.617220 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" Sep 08 10:02:37.903504 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:37.903433 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-d72f2"] Sep 08 10:02:37.984497 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:37.984402 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ktdpl" event={"ID":"2b7ee997-0723-4a47-90b5-805cd7721382","Type":"ContainerStarted","Data":"5a8d7fedaa5616dbf343638f4ab086793f4eedc9a67d51ea065ed3912a05d4a9"} Sep 08 10:02:37.986291 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:37.986256 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mvf7l" event={"ID":"334662a7-e44f-4aa9-8304-0a13dfed33e7","Type":"ContainerStarted","Data":"622e0afbee9f665e5f1847704fc3be7f52946c2860bb50ce85f084ab9e40abaf"} Sep 08 10:02:38.022954 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:02:38.022917 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01943a4b_bbca_418d_add3_d0169cd20bff.slice/crio-00094f33f8e0d158260a73ee52bb3cbece85ba9af2a3dfca9dec2e83115d30c9 WatchSource:0}: Error finding container 00094f33f8e0d158260a73ee52bb3cbece85ba9af2a3dfca9dec2e83115d30c9: Status 404 returned error can't find the container with id 00094f33f8e0d158260a73ee52bb3cbece85ba9af2a3dfca9dec2e83115d30c9 Sep 08 10:02:38.352324 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:38.352253 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:38.420893 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:38.420840 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:38.993892 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:38.993857 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jgstq" event={"ID":"01480054-d16a-4820-af73-7a493b53ead0","Type":"ContainerStarted","Data":"1239e01a1f0b10bc1c5f5cd3bf576e89ea8a4abd8c75b9153ba368dd0cb32f26"} Sep 08 10:02:38.994345 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:38.993917 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jgstq" event={"ID":"01480054-d16a-4820-af73-7a493b53ead0","Type":"ContainerStarted","Data":"ad1777a7a275fe4708486544036249b01e9335ff0c242a8db340320ab6446fd1"} Sep 08 10:02:38.996240 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:38.996208 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" event={"ID":"01943a4b-bbca-418d-add3-d0169cd20bff","Type":"ContainerStarted","Data":"00094f33f8e0d158260a73ee52bb3cbece85ba9af2a3dfca9dec2e83115d30c9"} Sep 08 10:02:39.013537 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:39.013492 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jgstq" podStartSLOduration=67.725767132 podStartE2EDuration="1m9.013475844s" totalImagesPullingTime="1.287708712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:39.012392688 +0000 UTC m=+68.801444492" watchObservedRunningTime="2026-09-08 10:02:39.013475844 +0000 UTC m=+68.802527648" Sep 08 10:02:39.089687 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:39.089655 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:39.190817 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:39.190787 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:40.336591 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:40.336561 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:40.337752 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:40.337565 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:40.344168 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:40.343990 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:41.007780 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:41.007419 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" event={"ID":"01943a4b-bbca-418d-add3-d0169cd20bff","Type":"ContainerStarted","Data":"d1a462ed31ae123ad4ebb4c48c65e8f1fb4f9bc5e3717652dc2ecbdeb850628c"} Sep 08 10:02:41.007780 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:41.007514 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" event={"ID":"01943a4b-bbca-418d-add3-d0169cd20bff","Type":"ContainerStarted","Data":"3df100437cd37be17730601bd3c9d41feefc166247fdb59163e30debded4e981"} Sep 08 10:02:41.013458 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:41.013431 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:02:41.043446 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:41.043390 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-d72f2" podStartSLOduration=2.902877558 podStartE2EDuration="5.043367267s" totalImagesPullingTime="2.140489709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:41.023930567 +0000 UTC m=+70.812982371" watchObservedRunningTime="2026-09-08 10:02:41.043367267 +0000 UTC m=+70.832419074" Sep 08 10:02:42.201140 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:42.201063 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:42.273717 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:42.273672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:42.335570 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:42.335532 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:42.808243 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:42.808180 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:43.068057 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.067981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:43.092551 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.092515 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bz65z"] Sep 08 10:02:43.097247 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.097220 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.100483 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.099352 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f9gjt\"" Sep 08 10:02:43.100483 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.099379 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 08 10:02:43.100483 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.099736 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 08 10:02:43.100483 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.099869 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 08 10:02:43.168029 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.167996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:43.180190 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.180159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2d18286-9927-4254-9ae7-0d1cd0470ae1-sys\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.180365 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.180200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-tls\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.180365 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.180247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5wzn\" (UniqueName: \"kubernetes.io/projected/e2d18286-9927-4254-9ae7-0d1cd0470ae1-kube-api-access-k5wzn\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.180365 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.180299 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-wtmp\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.180365 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.180324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2d18286-9927-4254-9ae7-0d1cd0470ae1-metrics-client-ca\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.180365 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.180366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-accelerators-collector-config\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.180651 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.180421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e2d18286-9927-4254-9ae7-0d1cd0470ae1-root\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.180651 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.180445 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.180651 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.180533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-textfile\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.264188 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.264153 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:43.281420 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.281393 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-wtmp\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.281588 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.281428 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2d18286-9927-4254-9ae7-0d1cd0470ae1-metrics-client-ca\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.281588 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.281461 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-accelerators-collector-config\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.281588 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.281491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e2d18286-9927-4254-9ae7-0d1cd0470ae1-root\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.281588 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.281506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.281588 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.281535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-textfile\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.281588 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.281564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2d18286-9927-4254-9ae7-0d1cd0470ae1-sys\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.281588 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.281583 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-tls\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.281947 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.281640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5wzn\" (UniqueName: \"kubernetes.io/projected/e2d18286-9927-4254-9ae7-0d1cd0470ae1-kube-api-access-k5wzn\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.281947 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.281879 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e2d18286-9927-4254-9ae7-0d1cd0470ae1-root\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.282037 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.282019 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-wtmp\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.282083 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.282040 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2d18286-9927-4254-9ae7-0d1cd0470ae1-sys\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.282221 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:43.282205 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 08 10:02:43.282292 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:43.282280 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-tls podName:e2d18286-9927-4254-9ae7-0d1cd0470ae1 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:43.782260292 +0000 UTC m=+73.571312079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-tls") pod "node-exporter-bz65z" (UID: "e2d18286-9927-4254-9ae7-0d1cd0470ae1") : secret "node-exporter-tls" not found Sep 08 10:02:43.282793 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.282772 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-textfile\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.283488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.283465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-accelerators-collector-config\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.283965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.283942 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2d18286-9927-4254-9ae7-0d1cd0470ae1-metrics-client-ca\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.284896 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.284874 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.294639 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.290910 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5wzn\" (UniqueName: \"kubernetes.io/projected/e2d18286-9927-4254-9ae7-0d1cd0470ae1-kube-api-access-k5wzn\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.370452 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.363996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:43.786657 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.786608 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-tls\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:43.789253 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:43.789225 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2d18286-9927-4254-9ae7-0d1cd0470ae1-node-exporter-tls\") pod \"node-exporter-bz65z\" (UID: \"e2d18286-9927-4254-9ae7-0d1cd0470ae1\") " pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:44.009581 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.009543 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bz65z" Sep 08 10:02:44.027233 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.027162 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mvf7l" event={"ID":"334662a7-e44f-4aa9-8304-0a13dfed33e7","Type":"ContainerStarted","Data":"7abb70565b89711f9f1c0e696d04c66a39e2b6b33426a55dabe92e434eaa78f3"} Sep 08 10:02:44.027470 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.027306 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:02:44.042349 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.042305 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mvf7l" podStartSLOduration=69.317393911 podStartE2EDuration="1m14.042290786s" totalImagesPullingTime="4.724896875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:44.041340897 +0000 UTC m=+73.830392725" watchObservedRunningTime="2026-09-08 10:02:44.042290786 +0000 UTC m=+73.831342590" Sep 08 10:02:44.142247 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.142216 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 10:02:44.168259 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.167470 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.168259 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.167922 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 10:02:44.170483 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.170460 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 08 10:02:44.171573 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.170916 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 08 10:02:44.171573 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.171171 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 08 10:02:44.175063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.173784 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 08 10:02:44.175063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.174076 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 08 10:02:44.175063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.174802 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 08 10:02:44.175541 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.175522 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 08 10:02:44.181655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.180246 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 08 10:02:44.181655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.180802 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 08 10:02:44.181655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.180878 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zg7lh\"" Sep 08 10:02:44.291092 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.291007 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-web-config\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.291092 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.291080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.291598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.291118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.291598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.291144 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3d01ee9c-4452-40e8-ba37-40dc870d36b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.291598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.291177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d01ee9c-4452-40e8-ba37-40dc870d36b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.291598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.291270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.291598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.291335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.291598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.291365 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.291598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.291391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d01ee9c-4452-40e8-ba37-40dc870d36b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.291598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.291423 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmp74\" (UniqueName: \"kubernetes.io/projected/3d01ee9c-4452-40e8-ba37-40dc870d36b2-kube-api-access-wmp74\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.291598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.291451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d01ee9c-4452-40e8-ba37-40dc870d36b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.291598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.291474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.291598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.291501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d01ee9c-4452-40e8-ba37-40dc870d36b2-config-out\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.392323 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.392278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-web-config\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.392500 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.392333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.392500 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.392367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.392500 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.392389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3d01ee9c-4452-40e8-ba37-40dc870d36b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.392500 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.392414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d01ee9c-4452-40e8-ba37-40dc870d36b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.392500 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.392444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.392500 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.392489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.392817 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.392511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.392817 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.392535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d01ee9c-4452-40e8-ba37-40dc870d36b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.392817 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.392565 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmp74\" (UniqueName: \"kubernetes.io/projected/3d01ee9c-4452-40e8-ba37-40dc870d36b2-kube-api-access-wmp74\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.392817 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.392591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d01ee9c-4452-40e8-ba37-40dc870d36b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.392817 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.392641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.392817 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.392670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d01ee9c-4452-40e8-ba37-40dc870d36b2-config-out\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.393456 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:02:44.393427 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d01ee9c-4452-40e8-ba37-40dc870d36b2-alertmanager-trusted-ca-bundle podName:3d01ee9c-4452-40e8-ba37-40dc870d36b2 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:44.893403527 +0000 UTC m=+74.682455324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d01ee9c-4452-40e8-ba37-40dc870d36b2-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "3d01ee9c-4452-40e8-ba37-40dc870d36b2") : configmap references non-existent config key: ca-bundle.crt Sep 08 10:02:44.394077 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.393732 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d01ee9c-4452-40e8-ba37-40dc870d36b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.394466 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.394438 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3d01ee9c-4452-40e8-ba37-40dc870d36b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.395761 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.395737 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.397052 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.397009 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-web-config\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.397576 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.397532 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.398369 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.398324 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d01ee9c-4452-40e8-ba37-40dc870d36b2-config-out\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.398493 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.398443 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.399234 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.399146 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d01ee9c-4452-40e8-ba37-40dc870d36b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.399234 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.399191 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.399339 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.399247 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.399807 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.399784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3d01ee9c-4452-40e8-ba37-40dc870d36b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.409858 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.409831 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmp74\" (UniqueName: \"kubernetes.io/projected/3d01ee9c-4452-40e8-ba37-40dc870d36b2-kube-api-access-wmp74\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.898163 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.898126 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d01ee9c-4452-40e8-ba37-40dc870d36b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:44.899060 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:44.899030 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d01ee9c-4452-40e8-ba37-40dc870d36b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3d01ee9c-4452-40e8-ba37-40dc870d36b2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:45.033161 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:45.033122 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bz65z" event={"ID":"e2d18286-9927-4254-9ae7-0d1cd0470ae1","Type":"ContainerStarted","Data":"5366eaee5b337c46f93590c11c9077b9be1dc9b59947c1f403071901d99764a1"} Sep 08 10:02:45.034943 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:45.034910 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ktdpl" event={"ID":"2b7ee997-0723-4a47-90b5-805cd7721382","Type":"ContainerStarted","Data":"b7dc966bab1df270c424db928f1d68f72e471a622b0b1620a518c2f80e68c3f2"} Sep 08 10:02:45.058695 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:45.058612 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ktdpl" podStartSLOduration=67.953977186 podStartE2EDuration="1m15.05859241s" totalImagesPullingTime="7.104615224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:45.057262531 +0000 UTC m=+74.846314339" watchObservedRunningTime="2026-09-08 10:02:45.05859241 +0000 UTC m=+74.847644215" Sep 08 10:02:45.094647 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:45.094595 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 10:02:45.120483 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:45.120447 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:45.182967 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:45.182935 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:45.384587 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:45.384558 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 10:02:45.385212 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:02:45.385185 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d01ee9c_4452_40e8_ba37_40dc870d36b2.slice/crio-8156815feefd13a777b98b3ec9bf3c2780e08ec70a143bb11341808f5cc73411 WatchSource:0}: Error finding container 8156815feefd13a777b98b3ec9bf3c2780e08ec70a143bb11341808f5cc73411: Status 404 returned error can't find the container with id 8156815feefd13a777b98b3ec9bf3c2780e08ec70a143bb11341808f5cc73411 Sep 08 10:02:46.018765 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:46.018544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:46.041506 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:46.041471 2576 generic.go:356] "Generic (PLEG): container finished" podID="e2d18286-9927-4254-9ae7-0d1cd0470ae1" containerID="d022be13137093e12d5cbc0e72c9c68680fb964cf18247fc0daed476313c57fb" exitCode=0 Sep 08 10:02:46.041656 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:46.041550 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bz65z" event={"ID":"e2d18286-9927-4254-9ae7-0d1cd0470ae1","Type":"ContainerDied","Data":"d022be13137093e12d5cbc0e72c9c68680fb964cf18247fc0daed476313c57fb"} Sep 08 10:02:46.043487 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:46.043451 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d01ee9c-4452-40e8-ba37-40dc870d36b2","Type":"ContainerStarted","Data":"8156815feefd13a777b98b3ec9bf3c2780e08ec70a143bb11341808f5cc73411"} Sep 08 10:02:46.191962 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:46.191935 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:46.528527 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:46.528489 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:46.628701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:46.628650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:47.049867 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:47.049828 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bz65z" event={"ID":"e2d18286-9927-4254-9ae7-0d1cd0470ae1","Type":"ContainerStarted","Data":"112b9a7b5e20594ce8d9229de2e57648dc43cff82943039b0fa4d01eeb968bf9"} Sep 08 10:02:47.050061 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:47.049896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bz65z" event={"ID":"e2d18286-9927-4254-9ae7-0d1cd0470ae1","Type":"ContainerStarted","Data":"2acd61065bde0feb30ed3ebd83402be20563b0ee50d0dc1e77f2caac218e7795"} Sep 08 10:02:47.075845 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:47.075790 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bz65z" podStartSLOduration=2.312846337 podStartE2EDuration="4.075770703s" totalImagesPullingTime="1.682991741s" totalInitContainerRuntime="79.932625ms" isStatefulPod=true podCreationTimestamp="2026-09-08 10:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:47.07400518 +0000 UTC m=+76.863057009" watchObservedRunningTime="2026-09-08 10:02:47.075770703 +0000 UTC m=+76.864822509" Sep 08 10:02:47.323186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:47.323096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:47.412070 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:47.412035 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:47.820291 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:47.820261 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:47.889235 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:47.889177 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:48.190817 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:48.190776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:48.272841 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:48.272802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:48.437294 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:48.437259 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:48.672710 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:48.672668 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:50.367017 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:50.366966 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:50.466890 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:50.466849 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:52.958199 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:52.958067 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6666db957f-c2qnq" Sep 08 10:02:53.088579 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:53.088525 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:55.089280 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:55.089194 2576 generic.go:356] "Generic (PLEG): container finished" podID="3d01ee9c-4452-40e8-ba37-40dc870d36b2" containerID="55009ccd2c9849730bcea375ccfea3038d9eed25cde7e6cf6335bbe65a4713ef" exitCode=0 Sep 08 10:02:55.089280 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:55.089235 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d01ee9c-4452-40e8-ba37-40dc870d36b2","Type":"ContainerDied","Data":"55009ccd2c9849730bcea375ccfea3038d9eed25cde7e6cf6335bbe65a4713ef"} Sep 08 10:02:55.091043 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:55.091018 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-qhcfl" event={"ID":"00a89ef4-4cc0-4f8c-b01f-b8dfa24e340a","Type":"ContainerStarted","Data":"fe4508e4ab079c83c7db529bc76ad354faa813084a1de9a4ba186fe231d688f7"} Sep 08 10:02:55.091253 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:55.091235 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-qhcfl" Sep 08 10:02:55.092816 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:55.092787 2576 patch_prober.go:28] interesting pod/downloads-85d98c998f-qhcfl container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" start-of-body= Sep 08 10:02:55.092907 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:55.092834 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-qhcfl" podUID="00a89ef4-4cc0-4f8c-b01f-b8dfa24e340a" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" Sep 08 10:02:55.132260 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:55.132202 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-qhcfl" podStartSLOduration=1.341397065 podStartE2EDuration="26.132182347s" totalImagesPullingTime="24.790785282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:55.130293586 +0000 UTC m=+84.919345391" watchObservedRunningTime="2026-09-08 10:02:55.132182347 +0000 UTC m=+84.921234151" Sep 08 10:02:56.113160 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:56.113124 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-qhcfl" Sep 08 10:02:56.167526 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:56.167480 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:56.270899 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:56.270856 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:57.095008 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:57.094928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:57.206089 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:57.206042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:57.296915 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:57.296878 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:02:59.113228 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:59.113187 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d01ee9c-4452-40e8-ba37-40dc870d36b2","Type":"ContainerStarted","Data":"a2cd5a018563e34526afe1637d2421e046b732da5ad16cc588f9e0dc5ca5355a"} Sep 08 10:02:59.113637 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:59.113266 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d01ee9c-4452-40e8-ba37-40dc870d36b2","Type":"ContainerStarted","Data":"2000278871a29d592a2d781ece4ac64bdffdd9767f8153360547c15e7a58b319"} Sep 08 10:02:59.113637 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:59.113305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d01ee9c-4452-40e8-ba37-40dc870d36b2","Type":"ContainerStarted","Data":"614c59871e6c5dad1716bcfa0ffa29eef68af42992bb76c6acc685e6c65ecd5f"} Sep 08 10:02:59.113637 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:59.113339 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d01ee9c-4452-40e8-ba37-40dc870d36b2","Type":"ContainerStarted","Data":"48168c7ceeacc90d217b0da43e7053894dc8cf8dc6574abd6009074e436c3457"} Sep 08 10:02:59.113637 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:02:59.113376 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d01ee9c-4452-40e8-ba37-40dc870d36b2","Type":"ContainerStarted","Data":"7b822c1f82d0d166b2fa2f7ec59c5c065fe247f9af23f06f050bdfaa86ee843d"} Sep 08 10:03:01.126066 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:01.126024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d01ee9c-4452-40e8-ba37-40dc870d36b2","Type":"ContainerStarted","Data":"e5219ab107b39bf3487cbdb24349c48272fd06d8bddbce83ac1e8262e85d57ca"} Sep 08 10:03:01.155917 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:01.155856 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.335096575 podStartE2EDuration="17.15583472s" totalImagesPullingTime="13.809356068s" totalInitContainerRuntime="11.382077ms" isStatefulPod=false podCreationTimestamp="2026-09-08 10:02:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:03:01.15404461 +0000 UTC m=+90.943096424" watchObservedRunningTime="2026-09-08 10:03:01.15583472 +0000 UTC m=+90.944886559" Sep 08 10:03:02.317389 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:02.317345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:03:02.417229 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:02.417176 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:03:03.338149 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:03.338109 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f7448cf6-v87wv"] Sep 08 10:03:03.433892 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:03.433857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:03:03.534251 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:03.534199 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:03:03.634680 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:03.634590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:03:06.167631 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:06.167597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:03:06.269505 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:06.269475 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:03:09.719669 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:09.719632 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:03:09.819829 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:09.819748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:03:15.037891 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:15.037861 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mvf7l" Sep 08 10:03:17.337116 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:17.337081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:03:17.783427 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:17.783394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:03:28.362916 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.362855 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f7448cf6-v87wv" podUID="a88ac405-fc3e-47c7-bb68-313c148de5ef" containerName="console" containerID="cri-o://ada2d3f20bffc1cbe509790d1ad2eb5a6d917670c1255f48c701d601b71a1091" gracePeriod=15 Sep 08 10:03:28.654508 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.654472 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:03:28.755172 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.755135 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pddpc\" (UniqueName: \"kubernetes.io/projected/a88ac405-fc3e-47c7-bb68-313c148de5ef-kube-api-access-pddpc\") pod \"a88ac405-fc3e-47c7-bb68-313c148de5ef\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " Sep 08 10:03:28.755172 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.755182 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-oauth-serving-cert\") pod \"a88ac405-fc3e-47c7-bb68-313c148de5ef\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " Sep 08 10:03:28.755410 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.755198 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-oauth-config\") pod \"a88ac405-fc3e-47c7-bb68-313c148de5ef\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " Sep 08 10:03:28.755410 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.755214 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-service-ca\") pod \"a88ac405-fc3e-47c7-bb68-313c148de5ef\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " Sep 08 10:03:28.755410 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.755242 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-config\") pod \"a88ac405-fc3e-47c7-bb68-313c148de5ef\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " Sep 08 10:03:28.755410 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.755277 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-serving-cert\") pod \"a88ac405-fc3e-47c7-bb68-313c148de5ef\" (UID: \"a88ac405-fc3e-47c7-bb68-313c148de5ef\") " Sep 08 10:03:28.755642 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.755592 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-service-ca" (OuterVolumeSpecName: "service-ca") pod "a88ac405-fc3e-47c7-bb68-313c148de5ef" (UID: "a88ac405-fc3e-47c7-bb68-313c148de5ef"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:03:28.755708 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.755647 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a88ac405-fc3e-47c7-bb68-313c148de5ef" (UID: "a88ac405-fc3e-47c7-bb68-313c148de5ef"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:03:28.755752 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.755713 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-config" (OuterVolumeSpecName: "console-config") pod "a88ac405-fc3e-47c7-bb68-313c148de5ef" (UID: "a88ac405-fc3e-47c7-bb68-313c148de5ef"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:03:28.757583 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.757558 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a88ac405-fc3e-47c7-bb68-313c148de5ef" (UID: "a88ac405-fc3e-47c7-bb68-313c148de5ef"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:03:28.757726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.757651 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88ac405-fc3e-47c7-bb68-313c148de5ef-kube-api-access-pddpc" (OuterVolumeSpecName: "kube-api-access-pddpc") pod "a88ac405-fc3e-47c7-bb68-313c148de5ef" (UID: "a88ac405-fc3e-47c7-bb68-313c148de5ef"). InnerVolumeSpecName "kube-api-access-pddpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:03:28.757726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.757659 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a88ac405-fc3e-47c7-bb68-313c148de5ef" (UID: "a88ac405-fc3e-47c7-bb68-313c148de5ef"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:03:28.856015 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.855984 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pddpc\" (UniqueName: \"kubernetes.io/projected/a88ac405-fc3e-47c7-bb68-313c148de5ef-kube-api-access-pddpc\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:03:28.856015 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.856010 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-oauth-serving-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:03:28.856015 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.856019 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-oauth-config\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:03:28.856234 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.856029 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-service-ca\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:03:28.856234 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.856038 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-config\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:03:28.856234 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:28.856045 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a88ac405-fc3e-47c7-bb68-313c148de5ef-console-serving-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:03:29.228851 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:29.228815 2576 generic.go:356] "Generic (PLEG): container finished" podID="a88ac405-fc3e-47c7-bb68-313c148de5ef" containerID="ada2d3f20bffc1cbe509790d1ad2eb5a6d917670c1255f48c701d601b71a1091" exitCode=0 Sep 08 10:03:29.229031 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:29.228858 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7448cf6-v87wv" Sep 08 10:03:29.229031 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:29.228869 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7448cf6-v87wv" event={"ID":"a88ac405-fc3e-47c7-bb68-313c148de5ef","Type":"ContainerDied","Data":"ada2d3f20bffc1cbe509790d1ad2eb5a6d917670c1255f48c701d601b71a1091"} Sep 08 10:03:29.229031 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:29.228917 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7448cf6-v87wv" event={"ID":"a88ac405-fc3e-47c7-bb68-313c148de5ef","Type":"ContainerDied","Data":"e9c82a8b9753d1238513b3e2f40af8eb9fd9fda35feb28de6956029a2bc3eb18"} Sep 08 10:03:29.229031 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:29.228949 2576 scope.go:117] "RemoveContainer" containerID="ada2d3f20bffc1cbe509790d1ad2eb5a6d917670c1255f48c701d601b71a1091" Sep 08 10:03:29.237422 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:29.237406 2576 scope.go:117] "RemoveContainer" containerID="ada2d3f20bffc1cbe509790d1ad2eb5a6d917670c1255f48c701d601b71a1091" Sep 08 10:03:29.237730 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:03:29.237710 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada2d3f20bffc1cbe509790d1ad2eb5a6d917670c1255f48c701d601b71a1091\": container with ID starting with ada2d3f20bffc1cbe509790d1ad2eb5a6d917670c1255f48c701d601b71a1091 not found: ID does not exist" containerID="ada2d3f20bffc1cbe509790d1ad2eb5a6d917670c1255f48c701d601b71a1091" Sep 08 10:03:29.237787 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:29.237738 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada2d3f20bffc1cbe509790d1ad2eb5a6d917670c1255f48c701d601b71a1091"} err="failed to get container status \"ada2d3f20bffc1cbe509790d1ad2eb5a6d917670c1255f48c701d601b71a1091\": rpc error: code = NotFound desc = could not find container \"ada2d3f20bffc1cbe509790d1ad2eb5a6d917670c1255f48c701d601b71a1091\": container with ID starting with ada2d3f20bffc1cbe509790d1ad2eb5a6d917670c1255f48c701d601b71a1091 not found: ID does not exist" Sep 08 10:03:29.248429 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:29.248405 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f7448cf6-v87wv"] Sep 08 10:03:29.252125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:29.252103 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f7448cf6-v87wv"] Sep 08 10:03:30.676668 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:30.676584 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a88ac405-fc3e-47c7-bb68-313c148de5ef" path="/var/lib/kubelet/pods/a88ac405-fc3e-47c7-bb68-313c148de5ef/volumes" Sep 08 10:03:56.281328 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:56.281288 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:03:59.208902 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:59.208864 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:03:59.293296 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:03:59.293258 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:04:03.506596 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:04:03.506557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:04:03.568452 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:04:03.568420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:04:03.797420 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:04:03.797337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:04:11.172878 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:04:11.172845 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:04:11.476810 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:04:11.476722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:04:13.774689 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:04:13.774657 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:04:13.876276 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:04:13.876244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:04:17.332608 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:04:17.332570 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:04:17.433233 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:04:17.433194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:04:22.361990 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:04:22.361951 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:04:49.947387 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:04:49.947306 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:04:50.005351 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:04:50.005322 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:05:33.543673 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:33.543585 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:05:33.644321 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:33.644291 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:05:34.281987 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.281949 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w"] Sep 08 10:05:34.282224 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.282211 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a88ac405-fc3e-47c7-bb68-313c148de5ef" containerName="console" Sep 08 10:05:34.282265 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.282225 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a88ac405-fc3e-47c7-bb68-313c148de5ef" containerName="console" Sep 08 10:05:34.282302 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.282292 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a88ac405-fc3e-47c7-bb68-313c148de5ef" containerName="console" Sep 08 10:05:34.284120 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.284105 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" Sep 08 10:05:34.286612 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.286590 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 10:05:34.286737 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.286690 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 10:05:34.287236 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.287222 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vlcsl\"" Sep 08 10:05:34.294070 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.294050 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w"] Sep 08 10:05:34.387460 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.387430 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/351d0240-3705-45c2-b0f2-0e47bb48f4b0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w\" (UID: \"351d0240-3705-45c2-b0f2-0e47bb48f4b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" Sep 08 10:05:34.387608 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.387475 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/351d0240-3705-45c2-b0f2-0e47bb48f4b0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w\" (UID: \"351d0240-3705-45c2-b0f2-0e47bb48f4b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" Sep 08 10:05:34.387608 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.387503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8xch\" (UniqueName: \"kubernetes.io/projected/351d0240-3705-45c2-b0f2-0e47bb48f4b0-kube-api-access-c8xch\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w\" (UID: \"351d0240-3705-45c2-b0f2-0e47bb48f4b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" Sep 08 10:05:34.488896 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.488856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/351d0240-3705-45c2-b0f2-0e47bb48f4b0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w\" (UID: \"351d0240-3705-45c2-b0f2-0e47bb48f4b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" Sep 08 10:05:34.489029 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.488920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/351d0240-3705-45c2-b0f2-0e47bb48f4b0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w\" (UID: \"351d0240-3705-45c2-b0f2-0e47bb48f4b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" Sep 08 10:05:34.489029 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.488962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8xch\" (UniqueName: \"kubernetes.io/projected/351d0240-3705-45c2-b0f2-0e47bb48f4b0-kube-api-access-c8xch\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w\" (UID: \"351d0240-3705-45c2-b0f2-0e47bb48f4b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" Sep 08 10:05:34.489252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.489232 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/351d0240-3705-45c2-b0f2-0e47bb48f4b0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w\" (UID: \"351d0240-3705-45c2-b0f2-0e47bb48f4b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" Sep 08 10:05:34.489338 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.489320 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/351d0240-3705-45c2-b0f2-0e47bb48f4b0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w\" (UID: \"351d0240-3705-45c2-b0f2-0e47bb48f4b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" Sep 08 10:05:34.497254 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.497231 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8xch\" (UniqueName: \"kubernetes.io/projected/351d0240-3705-45c2-b0f2-0e47bb48f4b0-kube-api-access-c8xch\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w\" (UID: \"351d0240-3705-45c2-b0f2-0e47bb48f4b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" Sep 08 10:05:34.593112 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.593025 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" Sep 08 10:05:34.860788 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:34.860709 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w"] Sep 08 10:05:34.864330 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:05:34.864295 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351d0240_3705_45c2_b0f2_0e47bb48f4b0.slice/crio-384fb045b8d3f746afec04c9a3be730ecde0a51bff56ec41644fbfe97a65d863 WatchSource:0}: Error finding container 384fb045b8d3f746afec04c9a3be730ecde0a51bff56ec41644fbfe97a65d863: Status 404 returned error can't find the container with id 384fb045b8d3f746afec04c9a3be730ecde0a51bff56ec41644fbfe97a65d863 Sep 08 10:05:35.680777 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:35.680744 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" event={"ID":"351d0240-3705-45c2-b0f2-0e47bb48f4b0","Type":"ContainerStarted","Data":"384fb045b8d3f746afec04c9a3be730ecde0a51bff56ec41644fbfe97a65d863"} Sep 08 10:05:41.707022 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:41.706983 2576 generic.go:356] "Generic (PLEG): container finished" podID="351d0240-3705-45c2-b0f2-0e47bb48f4b0" containerID="f2b04f84dafc01eaeb8a3179387bd2cd8c36d3c323a9e44484a0e6cb0ac1a0db" exitCode=0 Sep 08 10:05:41.707476 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:41.707076 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" event={"ID":"351d0240-3705-45c2-b0f2-0e47bb48f4b0","Type":"ContainerDied","Data":"f2b04f84dafc01eaeb8a3179387bd2cd8c36d3c323a9e44484a0e6cb0ac1a0db"} Sep 08 10:05:44.718455 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:44.718421 2576 generic.go:356] "Generic (PLEG): container finished" podID="351d0240-3705-45c2-b0f2-0e47bb48f4b0" containerID="d9facd3e54e6d20b3297d246b1d57808aeef798b120465a398821ba75f1ae602" exitCode=0 Sep 08 10:05:44.718827 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:44.718511 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" event={"ID":"351d0240-3705-45c2-b0f2-0e47bb48f4b0","Type":"ContainerDied","Data":"d9facd3e54e6d20b3297d246b1d57808aeef798b120465a398821ba75f1ae602"} Sep 08 10:05:53.752031 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:53.751938 2576 generic.go:356] "Generic (PLEG): container finished" podID="351d0240-3705-45c2-b0f2-0e47bb48f4b0" containerID="3a9a6fda15dc1bdf24c4fa49e535fb36d437d296c2a6f2bc60dceca3fbef33cf" exitCode=0 Sep 08 10:05:53.752031 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:53.752012 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" event={"ID":"351d0240-3705-45c2-b0f2-0e47bb48f4b0","Type":"ContainerDied","Data":"3a9a6fda15dc1bdf24c4fa49e535fb36d437d296c2a6f2bc60dceca3fbef33cf"} Sep 08 10:05:54.871070 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:54.871046 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" Sep 08 10:05:54.958194 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:54.958168 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8xch\" (UniqueName: \"kubernetes.io/projected/351d0240-3705-45c2-b0f2-0e47bb48f4b0-kube-api-access-c8xch\") pod \"351d0240-3705-45c2-b0f2-0e47bb48f4b0\" (UID: \"351d0240-3705-45c2-b0f2-0e47bb48f4b0\") " Sep 08 10:05:54.958310 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:54.958220 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/351d0240-3705-45c2-b0f2-0e47bb48f4b0-bundle\") pod \"351d0240-3705-45c2-b0f2-0e47bb48f4b0\" (UID: \"351d0240-3705-45c2-b0f2-0e47bb48f4b0\") " Sep 08 10:05:54.958310 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:54.958249 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/351d0240-3705-45c2-b0f2-0e47bb48f4b0-util\") pod \"351d0240-3705-45c2-b0f2-0e47bb48f4b0\" (UID: \"351d0240-3705-45c2-b0f2-0e47bb48f4b0\") " Sep 08 10:05:54.958908 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:54.958876 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351d0240-3705-45c2-b0f2-0e47bb48f4b0-bundle" (OuterVolumeSpecName: "bundle") pod "351d0240-3705-45c2-b0f2-0e47bb48f4b0" (UID: "351d0240-3705-45c2-b0f2-0e47bb48f4b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:05:54.960271 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:54.960235 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351d0240-3705-45c2-b0f2-0e47bb48f4b0-kube-api-access-c8xch" (OuterVolumeSpecName: "kube-api-access-c8xch") pod "351d0240-3705-45c2-b0f2-0e47bb48f4b0" (UID: "351d0240-3705-45c2-b0f2-0e47bb48f4b0"). InnerVolumeSpecName "kube-api-access-c8xch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:05:54.962172 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:54.962152 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351d0240-3705-45c2-b0f2-0e47bb48f4b0-util" (OuterVolumeSpecName: "util") pod "351d0240-3705-45c2-b0f2-0e47bb48f4b0" (UID: "351d0240-3705-45c2-b0f2-0e47bb48f4b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:05:55.059753 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:55.059692 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8xch\" (UniqueName: \"kubernetes.io/projected/351d0240-3705-45c2-b0f2-0e47bb48f4b0-kube-api-access-c8xch\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:05:55.059753 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:55.059716 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/351d0240-3705-45c2-b0f2-0e47bb48f4b0-bundle\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:05:55.059753 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:55.059726 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/351d0240-3705-45c2-b0f2-0e47bb48f4b0-util\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:05:55.760811 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:55.760766 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" event={"ID":"351d0240-3705-45c2-b0f2-0e47bb48f4b0","Type":"ContainerDied","Data":"384fb045b8d3f746afec04c9a3be730ecde0a51bff56ec41644fbfe97a65d863"} Sep 08 10:05:55.760811 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:55.760795 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v2m8w" Sep 08 10:05:55.760811 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:05:55.760819 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="384fb045b8d3f746afec04c9a3be730ecde0a51bff56ec41644fbfe97a65d863" Sep 08 10:06:02.029967 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:02.029897 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:02.134509 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:02.134453 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:03.973789 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:03.973744 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:04.078792 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:04.078750 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:06.354189 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.354116 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:06.476496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.476448 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:06.556418 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.556386 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n"] Sep 08 10:06:06.556658 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.556647 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351d0240-3705-45c2-b0f2-0e47bb48f4b0" containerName="util" Sep 08 10:06:06.556703 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.556660 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="351d0240-3705-45c2-b0f2-0e47bb48f4b0" containerName="util" Sep 08 10:06:06.556703 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.556678 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351d0240-3705-45c2-b0f2-0e47bb48f4b0" containerName="extract" Sep 08 10:06:06.556703 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.556684 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="351d0240-3705-45c2-b0f2-0e47bb48f4b0" containerName="extract" Sep 08 10:06:06.556703 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.556691 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351d0240-3705-45c2-b0f2-0e47bb48f4b0" containerName="pull" Sep 08 10:06:06.556703 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.556696 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="351d0240-3705-45c2-b0f2-0e47bb48f4b0" containerName="pull" Sep 08 10:06:06.560931 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.556738 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="351d0240-3705-45c2-b0f2-0e47bb48f4b0" containerName="extract" Sep 08 10:06:06.564660 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.564639 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" Sep 08 10:06:06.571055 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.570829 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 10:06:06.571227 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.571207 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 10:06:06.571884 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.571863 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vlcsl\"" Sep 08 10:06:06.582878 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.582848 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n"] Sep 08 10:06:06.665301 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.665258 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:06.744266 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.744236 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63a4ed1f-0148-496f-96e1-834dd326ce8e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n\" (UID: \"63a4ed1f-0148-496f-96e1-834dd326ce8e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" Sep 08 10:06:06.744266 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.744269 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9qj\" (UniqueName: \"kubernetes.io/projected/63a4ed1f-0148-496f-96e1-834dd326ce8e-kube-api-access-gr9qj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n\" (UID: \"63a4ed1f-0148-496f-96e1-834dd326ce8e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" Sep 08 10:06:06.744459 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.744293 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63a4ed1f-0148-496f-96e1-834dd326ce8e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n\" (UID: \"63a4ed1f-0148-496f-96e1-834dd326ce8e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" Sep 08 10:06:06.789131 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.789100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:06.845190 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.845165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63a4ed1f-0148-496f-96e1-834dd326ce8e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n\" (UID: \"63a4ed1f-0148-496f-96e1-834dd326ce8e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" Sep 08 10:06:06.845375 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.845200 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9qj\" (UniqueName: \"kubernetes.io/projected/63a4ed1f-0148-496f-96e1-834dd326ce8e-kube-api-access-gr9qj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n\" (UID: \"63a4ed1f-0148-496f-96e1-834dd326ce8e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" Sep 08 10:06:06.845375 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.845246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63a4ed1f-0148-496f-96e1-834dd326ce8e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n\" (UID: \"63a4ed1f-0148-496f-96e1-834dd326ce8e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" Sep 08 10:06:06.845686 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.845663 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63a4ed1f-0148-496f-96e1-834dd326ce8e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n\" (UID: \"63a4ed1f-0148-496f-96e1-834dd326ce8e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" Sep 08 10:06:06.845790 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.845704 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63a4ed1f-0148-496f-96e1-834dd326ce8e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n\" (UID: \"63a4ed1f-0148-496f-96e1-834dd326ce8e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" Sep 08 10:06:06.859628 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.859599 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9qj\" (UniqueName: \"kubernetes.io/projected/63a4ed1f-0148-496f-96e1-834dd326ce8e-kube-api-access-gr9qj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n\" (UID: \"63a4ed1f-0148-496f-96e1-834dd326ce8e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" Sep 08 10:06:06.873411 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:06.873390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" Sep 08 10:06:07.146693 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:07.146665 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n"] Sep 08 10:06:07.148646 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:06:07.148601 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a4ed1f_0148_496f_96e1_834dd326ce8e.slice/crio-31c06dc29d76d44c4198d204d4aba4d4812610ca8d4ae0b726562e2adb0b3285 WatchSource:0}: Error finding container 31c06dc29d76d44c4198d204d4aba4d4812610ca8d4ae0b726562e2adb0b3285: Status 404 returned error can't find the container with id 31c06dc29d76d44c4198d204d4aba4d4812610ca8d4ae0b726562e2adb0b3285 Sep 08 10:06:07.802201 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:07.802165 2576 generic.go:356] "Generic (PLEG): container finished" podID="63a4ed1f-0148-496f-96e1-834dd326ce8e" containerID="edb6daf0325e16b12c9fb6d8106899ceef5751ce396304665a90ecb677920c82" exitCode=0 Sep 08 10:06:07.802575 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:07.802255 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" event={"ID":"63a4ed1f-0148-496f-96e1-834dd326ce8e","Type":"ContainerDied","Data":"edb6daf0325e16b12c9fb6d8106899ceef5751ce396304665a90ecb677920c82"} Sep 08 10:06:07.802575 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:07.802303 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" event={"ID":"63a4ed1f-0148-496f-96e1-834dd326ce8e","Type":"ContainerStarted","Data":"31c06dc29d76d44c4198d204d4aba4d4812610ca8d4ae0b726562e2adb0b3285"} Sep 08 10:06:09.062110 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:09.062067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:09.236070 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:09.236038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:10.816070 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:10.816028 2576 generic.go:356] "Generic (PLEG): container finished" podID="63a4ed1f-0148-496f-96e1-834dd326ce8e" containerID="6c4d44a17493270b3648d2a642ae3b49c88bed6cea17b74affcd8e0180a52cda" exitCode=0 Sep 08 10:06:10.816434 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:10.816113 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" event={"ID":"63a4ed1f-0148-496f-96e1-834dd326ce8e","Type":"ContainerDied","Data":"6c4d44a17493270b3648d2a642ae3b49c88bed6cea17b74affcd8e0180a52cda"} Sep 08 10:06:11.375593 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:11.375558 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:11.464177 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:11.464145 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:11.822162 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:11.822125 2576 generic.go:356] "Generic (PLEG): container finished" podID="63a4ed1f-0148-496f-96e1-834dd326ce8e" containerID="c704892578403548067cb464f3b93edcecf83491b0b161ab734be483f84ad1a4" exitCode=0 Sep 08 10:06:11.822552 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:11.822208 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" event={"ID":"63a4ed1f-0148-496f-96e1-834dd326ce8e","Type":"ContainerDied","Data":"c704892578403548067cb464f3b93edcecf83491b0b161ab734be483f84ad1a4"} Sep 08 10:06:12.946918 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:12.946894 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" Sep 08 10:06:12.984040 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:12.984015 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr9qj\" (UniqueName: \"kubernetes.io/projected/63a4ed1f-0148-496f-96e1-834dd326ce8e-kube-api-access-gr9qj\") pod \"63a4ed1f-0148-496f-96e1-834dd326ce8e\" (UID: \"63a4ed1f-0148-496f-96e1-834dd326ce8e\") " Sep 08 10:06:12.984196 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:12.984070 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63a4ed1f-0148-496f-96e1-834dd326ce8e-bundle\") pod \"63a4ed1f-0148-496f-96e1-834dd326ce8e\" (UID: \"63a4ed1f-0148-496f-96e1-834dd326ce8e\") " Sep 08 10:06:12.984196 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:12.984124 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63a4ed1f-0148-496f-96e1-834dd326ce8e-util\") pod \"63a4ed1f-0148-496f-96e1-834dd326ce8e\" (UID: \"63a4ed1f-0148-496f-96e1-834dd326ce8e\") " Sep 08 10:06:12.984471 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:12.984445 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a4ed1f-0148-496f-96e1-834dd326ce8e-bundle" (OuterVolumeSpecName: "bundle") pod "63a4ed1f-0148-496f-96e1-834dd326ce8e" (UID: "63a4ed1f-0148-496f-96e1-834dd326ce8e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:06:12.986292 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:12.986267 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a4ed1f-0148-496f-96e1-834dd326ce8e-kube-api-access-gr9qj" (OuterVolumeSpecName: "kube-api-access-gr9qj") pod "63a4ed1f-0148-496f-96e1-834dd326ce8e" (UID: "63a4ed1f-0148-496f-96e1-834dd326ce8e"). InnerVolumeSpecName "kube-api-access-gr9qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:06:12.988612 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:12.988589 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a4ed1f-0148-496f-96e1-834dd326ce8e-util" (OuterVolumeSpecName: "util") pod "63a4ed1f-0148-496f-96e1-834dd326ce8e" (UID: "63a4ed1f-0148-496f-96e1-834dd326ce8e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:06:13.084912 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:13.084812 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63a4ed1f-0148-496f-96e1-834dd326ce8e-util\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:06:13.084912 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:13.084857 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9qj\" (UniqueName: \"kubernetes.io/projected/63a4ed1f-0148-496f-96e1-834dd326ce8e-kube-api-access-gr9qj\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:06:13.084912 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:13.084871 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63a4ed1f-0148-496f-96e1-834dd326ce8e-bundle\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:06:13.833792 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:13.833762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" event={"ID":"63a4ed1f-0148-496f-96e1-834dd326ce8e","Type":"ContainerDied","Data":"31c06dc29d76d44c4198d204d4aba4d4812610ca8d4ae0b726562e2adb0b3285"} Sep 08 10:06:13.833792 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:13.833781 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mx6n" Sep 08 10:06:13.833962 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:13.833806 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31c06dc29d76d44c4198d204d4aba4d4812610ca8d4ae0b726562e2adb0b3285" Sep 08 10:06:14.574075 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.574038 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zlr9b"] Sep 08 10:06:14.574440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.574290 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63a4ed1f-0148-496f-96e1-834dd326ce8e" containerName="pull" Sep 08 10:06:14.574440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.574300 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63a4ed1f-0148-496f-96e1-834dd326ce8e" containerName="pull" Sep 08 10:06:14.574440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.574313 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63a4ed1f-0148-496f-96e1-834dd326ce8e" containerName="extract" Sep 08 10:06:14.574440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.574319 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63a4ed1f-0148-496f-96e1-834dd326ce8e" containerName="extract" Sep 08 10:06:14.574440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.574326 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63a4ed1f-0148-496f-96e1-834dd326ce8e" containerName="util" Sep 08 10:06:14.574440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.574331 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63a4ed1f-0148-496f-96e1-834dd326ce8e" containerName="util" Sep 08 10:06:14.574440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.574378 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="63a4ed1f-0148-496f-96e1-834dd326ce8e" containerName="extract" Sep 08 10:06:14.578195 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.578179 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-zlr9b" Sep 08 10:06:14.580703 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.580680 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 08 10:06:14.580829 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.580684 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 08 10:06:14.581318 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.581304 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-cl8h6\"" Sep 08 10:06:14.586833 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.586813 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zlr9b"] Sep 08 10:06:14.595142 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.595124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgphw\" (UniqueName: \"kubernetes.io/projected/30236f1c-75a2-40ad-a6d3-860137a1db30-kube-api-access-kgphw\") pod \"cert-manager-webhook-6486ffd997-zlr9b\" (UID: \"30236f1c-75a2-40ad-a6d3-860137a1db30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zlr9b" Sep 08 10:06:14.595229 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.595177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30236f1c-75a2-40ad-a6d3-860137a1db30-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zlr9b\" (UID: \"30236f1c-75a2-40ad-a6d3-860137a1db30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zlr9b" Sep 08 10:06:14.696412 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.696383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30236f1c-75a2-40ad-a6d3-860137a1db30-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zlr9b\" (UID: \"30236f1c-75a2-40ad-a6d3-860137a1db30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zlr9b" Sep 08 10:06:14.696575 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.696424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgphw\" (UniqueName: \"kubernetes.io/projected/30236f1c-75a2-40ad-a6d3-860137a1db30-kube-api-access-kgphw\") pod \"cert-manager-webhook-6486ffd997-zlr9b\" (UID: \"30236f1c-75a2-40ad-a6d3-860137a1db30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zlr9b" Sep 08 10:06:14.704925 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.704896 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30236f1c-75a2-40ad-a6d3-860137a1db30-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zlr9b\" (UID: \"30236f1c-75a2-40ad-a6d3-860137a1db30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zlr9b" Sep 08 10:06:14.705103 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.705083 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgphw\" (UniqueName: \"kubernetes.io/projected/30236f1c-75a2-40ad-a6d3-860137a1db30-kube-api-access-kgphw\") pod \"cert-manager-webhook-6486ffd997-zlr9b\" (UID: \"30236f1c-75a2-40ad-a6d3-860137a1db30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zlr9b" Sep 08 10:06:14.887698 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:14.887572 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-zlr9b" Sep 08 10:06:15.157234 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:15.157197 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zlr9b"] Sep 08 10:06:15.160006 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:06:15.159975 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30236f1c_75a2_40ad_a6d3_860137a1db30.slice/crio-fa6e4e3b1ee1ec6c9f6f0e552732b971f363c9db163b2600fdaf7de05db392f2 WatchSource:0}: Error finding container fa6e4e3b1ee1ec6c9f6f0e552732b971f363c9db163b2600fdaf7de05db392f2: Status 404 returned error can't find the container with id fa6e4e3b1ee1ec6c9f6f0e552732b971f363c9db163b2600fdaf7de05db392f2 Sep 08 10:06:15.841752 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:15.841714 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-zlr9b" event={"ID":"30236f1c-75a2-40ad-a6d3-860137a1db30","Type":"ContainerStarted","Data":"fa6e4e3b1ee1ec6c9f6f0e552732b971f363c9db163b2600fdaf7de05db392f2"} Sep 08 10:06:17.304393 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:17.304351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:17.516534 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:17.516499 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-85vwm"] Sep 08 10:06:17.521726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:17.521706 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-85vwm" Sep 08 10:06:17.528029 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:17.528008 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6q65w\"" Sep 08 10:06:17.537808 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:17.537784 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-85vwm"] Sep 08 10:06:17.616677 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:17.616563 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz77m\" (UniqueName: \"kubernetes.io/projected/9adfb284-8d0a-4fd9-9ad1-39610ca4a35d-kube-api-access-lz77m\") pod \"cert-manager-55b4d954d5-85vwm\" (UID: \"9adfb284-8d0a-4fd9-9ad1-39610ca4a35d\") " pod="cert-manager/cert-manager-55b4d954d5-85vwm" Sep 08 10:06:17.616824 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:17.616677 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9adfb284-8d0a-4fd9-9ad1-39610ca4a35d-bound-sa-token\") pod \"cert-manager-55b4d954d5-85vwm\" (UID: \"9adfb284-8d0a-4fd9-9ad1-39610ca4a35d\") " pod="cert-manager/cert-manager-55b4d954d5-85vwm" Sep 08 10:06:17.717252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:17.717219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz77m\" (UniqueName: \"kubernetes.io/projected/9adfb284-8d0a-4fd9-9ad1-39610ca4a35d-kube-api-access-lz77m\") pod \"cert-manager-55b4d954d5-85vwm\" (UID: \"9adfb284-8d0a-4fd9-9ad1-39610ca4a35d\") " pod="cert-manager/cert-manager-55b4d954d5-85vwm" Sep 08 10:06:17.717397 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:17.717280 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9adfb284-8d0a-4fd9-9ad1-39610ca4a35d-bound-sa-token\") pod \"cert-manager-55b4d954d5-85vwm\" (UID: \"9adfb284-8d0a-4fd9-9ad1-39610ca4a35d\") " pod="cert-manager/cert-manager-55b4d954d5-85vwm" Sep 08 10:06:17.725641 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:17.725593 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9adfb284-8d0a-4fd9-9ad1-39610ca4a35d-bound-sa-token\") pod \"cert-manager-55b4d954d5-85vwm\" (UID: \"9adfb284-8d0a-4fd9-9ad1-39610ca4a35d\") " pod="cert-manager/cert-manager-55b4d954d5-85vwm" Sep 08 10:06:17.725738 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:17.725672 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz77m\" (UniqueName: \"kubernetes.io/projected/9adfb284-8d0a-4fd9-9ad1-39610ca4a35d-kube-api-access-lz77m\") pod \"cert-manager-55b4d954d5-85vwm\" (UID: \"9adfb284-8d0a-4fd9-9ad1-39610ca4a35d\") " pod="cert-manager/cert-manager-55b4d954d5-85vwm" Sep 08 10:06:17.830514 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:17.830480 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-85vwm" Sep 08 10:06:18.101159 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:18.101126 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-85vwm"] Sep 08 10:06:18.103885 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:06:18.103859 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9adfb284_8d0a_4fd9_9ad1_39610ca4a35d.slice/crio-4a3be52090ef1f6e56400f109ed323b42ab235ceb12565b234655d269e6dc6a9 WatchSource:0}: Error finding container 4a3be52090ef1f6e56400f109ed323b42ab235ceb12565b234655d269e6dc6a9: Status 404 returned error can't find the container with id 4a3be52090ef1f6e56400f109ed323b42ab235ceb12565b234655d269e6dc6a9 Sep 08 10:06:18.854837 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:18.854797 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-85vwm" event={"ID":"9adfb284-8d0a-4fd9-9ad1-39610ca4a35d","Type":"ContainerStarted","Data":"4a3be52090ef1f6e56400f109ed323b42ab235ceb12565b234655d269e6dc6a9"} Sep 08 10:06:21.176770 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:21.176721 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:21.269241 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:21.269183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:22.872454 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:22.872419 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-85vwm" event={"ID":"9adfb284-8d0a-4fd9-9ad1-39610ca4a35d","Type":"ContainerStarted","Data":"ec3095caea3e829dbd64c861977d01fb8066c20e15c54ec650100aeac003a737"} Sep 08 10:06:22.874024 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:22.874003 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-zlr9b" event={"ID":"30236f1c-75a2-40ad-a6d3-860137a1db30","Type":"ContainerStarted","Data":"b9ca0680042966a0fdaa268f3e4846d6a799a78eb635050db65e87b3bdcd1370"} Sep 08 10:06:22.874129 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:22.874115 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-zlr9b" Sep 08 10:06:22.887976 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:22.887932 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-85vwm" podStartSLOduration=2.127104314 podStartE2EDuration="5.88791845s" totalImagesPullingTime="3.760814136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:06:22.886268181 +0000 UTC m=+292.675319996" watchObservedRunningTime="2026-09-08 10:06:22.88791845 +0000 UTC m=+292.676970253" Sep 08 10:06:22.904451 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:22.904410 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-zlr9b" podStartSLOduration=2.193073299 podStartE2EDuration="8.904393408s" totalImagesPullingTime="6.711320109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:06:22.904024284 +0000 UTC m=+292.693076091" watchObservedRunningTime="2026-09-08 10:06:22.904393408 +0000 UTC m=+292.693445212" Sep 08 10:06:27.834476 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:27.834442 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:28.880066 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:28.880038 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-zlr9b" Sep 08 10:06:30.572764 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.572730 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7"] Sep 08 10:06:30.575250 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.575230 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" Sep 08 10:06:30.577671 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.577647 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 10:06:30.577778 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.577752 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 10:06:30.578419 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.578401 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vlcsl\"" Sep 08 10:06:30.584269 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.584243 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7"] Sep 08 10:06:30.605719 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.605700 2576 kubelet.go:1648] "Image garbage collection succeeded" Sep 08 10:06:30.606973 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.606958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dpm5\" (UniqueName: \"kubernetes.io/projected/400b768d-f98d-4547-b24c-fdce416896b6-kube-api-access-2dpm5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7\" (UID: \"400b768d-f98d-4547-b24c-fdce416896b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" Sep 08 10:06:30.607028 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.606985 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400b768d-f98d-4547-b24c-fdce416896b6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7\" (UID: \"400b768d-f98d-4547-b24c-fdce416896b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" Sep 08 10:06:30.607028 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.607013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400b768d-f98d-4547-b24c-fdce416896b6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7\" (UID: \"400b768d-f98d-4547-b24c-fdce416896b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" Sep 08 10:06:30.707384 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.707356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400b768d-f98d-4547-b24c-fdce416896b6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7\" (UID: \"400b768d-f98d-4547-b24c-fdce416896b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" Sep 08 10:06:30.708518 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.707472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dpm5\" (UniqueName: \"kubernetes.io/projected/400b768d-f98d-4547-b24c-fdce416896b6-kube-api-access-2dpm5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7\" (UID: \"400b768d-f98d-4547-b24c-fdce416896b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" Sep 08 10:06:30.708518 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.707504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400b768d-f98d-4547-b24c-fdce416896b6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7\" (UID: \"400b768d-f98d-4547-b24c-fdce416896b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" Sep 08 10:06:30.708518 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.707779 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400b768d-f98d-4547-b24c-fdce416896b6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7\" (UID: \"400b768d-f98d-4547-b24c-fdce416896b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" Sep 08 10:06:30.708518 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.707862 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400b768d-f98d-4547-b24c-fdce416896b6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7\" (UID: \"400b768d-f98d-4547-b24c-fdce416896b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" Sep 08 10:06:30.717416 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.717391 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dpm5\" (UniqueName: \"kubernetes.io/projected/400b768d-f98d-4547-b24c-fdce416896b6-kube-api-access-2dpm5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7\" (UID: \"400b768d-f98d-4547-b24c-fdce416896b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" Sep 08 10:06:30.885967 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:30.885905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" Sep 08 10:06:31.155460 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:31.155435 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7"] Sep 08 10:06:31.157463 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:06:31.157435 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400b768d_f98d_4547_b24c_fdce416896b6.slice/crio-f27b79635ad184997a4f82cde1d10c2e8f77d78c7aee1110a9e8f16dcd35d0c0 WatchSource:0}: Error finding container f27b79635ad184997a4f82cde1d10c2e8f77d78c7aee1110a9e8f16dcd35d0c0: Status 404 returned error can't find the container with id f27b79635ad184997a4f82cde1d10c2e8f77d78c7aee1110a9e8f16dcd35d0c0 Sep 08 10:06:31.908800 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:31.908756 2576 generic.go:356] "Generic (PLEG): container finished" podID="400b768d-f98d-4547-b24c-fdce416896b6" containerID="084d44bba6fbd3e04b71425e2bec6a228c681dbf9ca76c372981b6c4bd8b0fa9" exitCode=0 Sep 08 10:06:31.909163 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:31.908826 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" event={"ID":"400b768d-f98d-4547-b24c-fdce416896b6","Type":"ContainerDied","Data":"084d44bba6fbd3e04b71425e2bec6a228c681dbf9ca76c372981b6c4bd8b0fa9"} Sep 08 10:06:31.909163 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:31.908901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" event={"ID":"400b768d-f98d-4547-b24c-fdce416896b6","Type":"ContainerStarted","Data":"f27b79635ad184997a4f82cde1d10c2e8f77d78c7aee1110a9e8f16dcd35d0c0"} Sep 08 10:06:31.909343 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:31.909328 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 10:06:32.914525 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:32.914494 2576 generic.go:356] "Generic (PLEG): container finished" podID="400b768d-f98d-4547-b24c-fdce416896b6" containerID="edb7286f4191d5f791e702b2b8ebae8220eb5c709cb1d7f28be9595e95512b52" exitCode=0 Sep 08 10:06:32.914876 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:32.914578 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" event={"ID":"400b768d-f98d-4547-b24c-fdce416896b6","Type":"ContainerDied","Data":"edb7286f4191d5f791e702b2b8ebae8220eb5c709cb1d7f28be9595e95512b52"} Sep 08 10:06:33.908936 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:33.908903 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:33.921160 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:33.921134 2576 generic.go:356] "Generic (PLEG): container finished" podID="400b768d-f98d-4547-b24c-fdce416896b6" containerID="d9fdf3792d7fd0a646b66b25e511b734369016ea44a4fa8ae1002c8c01966617" exitCode=0 Sep 08 10:06:33.921472 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:33.921203 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" event={"ID":"400b768d-f98d-4547-b24c-fdce416896b6","Type":"ContainerDied","Data":"d9fdf3792d7fd0a646b66b25e511b734369016ea44a4fa8ae1002c8c01966617"} Sep 08 10:06:35.044179 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:35.044154 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" Sep 08 10:06:35.140140 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:35.140115 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dpm5\" (UniqueName: \"kubernetes.io/projected/400b768d-f98d-4547-b24c-fdce416896b6-kube-api-access-2dpm5\") pod \"400b768d-f98d-4547-b24c-fdce416896b6\" (UID: \"400b768d-f98d-4547-b24c-fdce416896b6\") " Sep 08 10:06:35.140254 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:35.140148 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400b768d-f98d-4547-b24c-fdce416896b6-util\") pod \"400b768d-f98d-4547-b24c-fdce416896b6\" (UID: \"400b768d-f98d-4547-b24c-fdce416896b6\") " Sep 08 10:06:35.140254 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:35.140170 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400b768d-f98d-4547-b24c-fdce416896b6-bundle\") pod \"400b768d-f98d-4547-b24c-fdce416896b6\" (UID: \"400b768d-f98d-4547-b24c-fdce416896b6\") " Sep 08 10:06:35.140878 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:35.140855 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400b768d-f98d-4547-b24c-fdce416896b6-bundle" (OuterVolumeSpecName: "bundle") pod "400b768d-f98d-4547-b24c-fdce416896b6" (UID: "400b768d-f98d-4547-b24c-fdce416896b6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:06:35.142039 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:35.142015 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400b768d-f98d-4547-b24c-fdce416896b6-kube-api-access-2dpm5" (OuterVolumeSpecName: "kube-api-access-2dpm5") pod "400b768d-f98d-4547-b24c-fdce416896b6" (UID: "400b768d-f98d-4547-b24c-fdce416896b6"). InnerVolumeSpecName "kube-api-access-2dpm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:06:35.145675 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:35.145535 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400b768d-f98d-4547-b24c-fdce416896b6-util" (OuterVolumeSpecName: "util") pod "400b768d-f98d-4547-b24c-fdce416896b6" (UID: "400b768d-f98d-4547-b24c-fdce416896b6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:06:35.241563 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:35.241497 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dpm5\" (UniqueName: \"kubernetes.io/projected/400b768d-f98d-4547-b24c-fdce416896b6-kube-api-access-2dpm5\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:06:35.241563 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:35.241528 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400b768d-f98d-4547-b24c-fdce416896b6-util\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:06:35.241563 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:35.241538 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400b768d-f98d-4547-b24c-fdce416896b6-bundle\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:06:35.930059 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:35.930028 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" event={"ID":"400b768d-f98d-4547-b24c-fdce416896b6","Type":"ContainerDied","Data":"f27b79635ad184997a4f82cde1d10c2e8f77d78c7aee1110a9e8f16dcd35d0c0"} Sep 08 10:06:35.930059 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:35.930057 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzgh7" Sep 08 10:06:35.930248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:35.930072 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f27b79635ad184997a4f82cde1d10c2e8f77d78c7aee1110a9e8f16dcd35d0c0" Sep 08 10:06:41.193114 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.193079 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx"] Sep 08 10:06:41.193481 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.193338 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400b768d-f98d-4547-b24c-fdce416896b6" containerName="pull" Sep 08 10:06:41.193481 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.193349 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="400b768d-f98d-4547-b24c-fdce416896b6" containerName="pull" Sep 08 10:06:41.193481 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.193361 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400b768d-f98d-4547-b24c-fdce416896b6" containerName="util" Sep 08 10:06:41.193481 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.193367 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="400b768d-f98d-4547-b24c-fdce416896b6" containerName="util" Sep 08 10:06:41.193481 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.193376 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400b768d-f98d-4547-b24c-fdce416896b6" containerName="extract" Sep 08 10:06:41.193481 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.193382 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="400b768d-f98d-4547-b24c-fdce416896b6" containerName="extract" Sep 08 10:06:41.193481 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.193423 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="400b768d-f98d-4547-b24c-fdce416896b6" containerName="extract" Sep 08 10:06:41.195651 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.195608 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" Sep 08 10:06:41.197969 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.197948 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 10:06:41.198088 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.198025 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 10:06:41.198836 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.198818 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vlcsl\"" Sep 08 10:06:41.207082 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.207061 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx"] Sep 08 10:06:41.286843 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.286814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7316bd8c-b4e8-4642-9722-8b44c7744e67-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx\" (UID: \"7316bd8c-b4e8-4642-9722-8b44c7744e67\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" Sep 08 10:06:41.287006 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.286873 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s8zf\" (UniqueName: \"kubernetes.io/projected/7316bd8c-b4e8-4642-9722-8b44c7744e67-kube-api-access-8s8zf\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx\" (UID: \"7316bd8c-b4e8-4642-9722-8b44c7744e67\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" Sep 08 10:06:41.287006 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.286936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7316bd8c-b4e8-4642-9722-8b44c7744e67-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx\" (UID: \"7316bd8c-b4e8-4642-9722-8b44c7744e67\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" Sep 08 10:06:41.387781 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.387753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s8zf\" (UniqueName: \"kubernetes.io/projected/7316bd8c-b4e8-4642-9722-8b44c7744e67-kube-api-access-8s8zf\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx\" (UID: \"7316bd8c-b4e8-4642-9722-8b44c7744e67\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" Sep 08 10:06:41.387909 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.387788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7316bd8c-b4e8-4642-9722-8b44c7744e67-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx\" (UID: \"7316bd8c-b4e8-4642-9722-8b44c7744e67\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" Sep 08 10:06:41.387909 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.387846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7316bd8c-b4e8-4642-9722-8b44c7744e67-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx\" (UID: \"7316bd8c-b4e8-4642-9722-8b44c7744e67\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" Sep 08 10:06:41.388230 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.388210 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7316bd8c-b4e8-4642-9722-8b44c7744e67-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx\" (UID: \"7316bd8c-b4e8-4642-9722-8b44c7744e67\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" Sep 08 10:06:41.388266 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.388223 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7316bd8c-b4e8-4642-9722-8b44c7744e67-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx\" (UID: \"7316bd8c-b4e8-4642-9722-8b44c7744e67\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" Sep 08 10:06:41.396483 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.396456 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s8zf\" (UniqueName: \"kubernetes.io/projected/7316bd8c-b4e8-4642-9722-8b44c7744e67-kube-api-access-8s8zf\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx\" (UID: \"7316bd8c-b4e8-4642-9722-8b44c7744e67\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" Sep 08 10:06:41.504461 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.504395 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" Sep 08 10:06:41.782167 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.782139 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx"] Sep 08 10:06:41.784654 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:06:41.784605 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7316bd8c_b4e8_4642_9722_8b44c7744e67.slice/crio-78328bc5f226aebee6879781bdab59aaa86f1cc9163a106a2bdb21678b739a5a WatchSource:0}: Error finding container 78328bc5f226aebee6879781bdab59aaa86f1cc9163a106a2bdb21678b739a5a: Status 404 returned error can't find the container with id 78328bc5f226aebee6879781bdab59aaa86f1cc9163a106a2bdb21678b739a5a Sep 08 10:06:41.955356 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.955326 2576 generic.go:356] "Generic (PLEG): container finished" podID="7316bd8c-b4e8-4642-9722-8b44c7744e67" containerID="69f45ce861dd507f1f2dee662366223788053e8b93fc8da4bb4d43a95f8a9b53" exitCode=0 Sep 08 10:06:41.955468 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.955401 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" event={"ID":"7316bd8c-b4e8-4642-9722-8b44c7744e67","Type":"ContainerDied","Data":"69f45ce861dd507f1f2dee662366223788053e8b93fc8da4bb4d43a95f8a9b53"} Sep 08 10:06:41.955468 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:41.955447 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" event={"ID":"7316bd8c-b4e8-4642-9722-8b44c7744e67","Type":"ContainerStarted","Data":"78328bc5f226aebee6879781bdab59aaa86f1cc9163a106a2bdb21678b739a5a"} Sep 08 10:06:42.307795 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:42.307761 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:42.413930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:42.413900 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:42.511740 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:42.511710 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:42.935905 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:42.935872 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:43.579102 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.579069 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph"] Sep 08 10:06:43.581375 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.581357 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.583759 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.583738 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 08 10:06:43.584411 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.584397 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 08 10:06:43.584645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.584607 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 08 10:06:43.584781 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.584689 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nb2g9\"" Sep 08 10:06:43.585126 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.585112 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 08 10:06:43.605250 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.605225 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph"] Sep 08 10:06:43.607699 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.607672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/17f6623a-31b9-4352-b713-ecbb082847f6-manifests\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-xvsph\" (UID: \"17f6623a-31b9-4352-b713-ecbb082847f6\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.607787 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.607708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znztk\" (UniqueName: \"kubernetes.io/projected/17f6623a-31b9-4352-b713-ecbb082847f6-kube-api-access-znztk\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-xvsph\" (UID: \"17f6623a-31b9-4352-b713-ecbb082847f6\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.607787 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.607771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/17f6623a-31b9-4352-b713-ecbb082847f6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-xvsph\" (UID: \"17f6623a-31b9-4352-b713-ecbb082847f6\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.607892 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.607823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17f6623a-31b9-4352-b713-ecbb082847f6-tmp\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-xvsph\" (UID: \"17f6623a-31b9-4352-b713-ecbb082847f6\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.607925 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.607887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17f6623a-31b9-4352-b713-ecbb082847f6-webhook-cert\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-xvsph\" (UID: \"17f6623a-31b9-4352-b713-ecbb082847f6\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.708649 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.708600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17f6623a-31b9-4352-b713-ecbb082847f6-tmp\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-xvsph\" (UID: \"17f6623a-31b9-4352-b713-ecbb082847f6\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.708772 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.708706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17f6623a-31b9-4352-b713-ecbb082847f6-webhook-cert\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-xvsph\" (UID: \"17f6623a-31b9-4352-b713-ecbb082847f6\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.709078 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.708853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/17f6623a-31b9-4352-b713-ecbb082847f6-manifests\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-xvsph\" (UID: \"17f6623a-31b9-4352-b713-ecbb082847f6\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.709078 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.708885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znztk\" (UniqueName: \"kubernetes.io/projected/17f6623a-31b9-4352-b713-ecbb082847f6-kube-api-access-znztk\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-xvsph\" (UID: \"17f6623a-31b9-4352-b713-ecbb082847f6\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.709078 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.708950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/17f6623a-31b9-4352-b713-ecbb082847f6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-xvsph\" (UID: \"17f6623a-31b9-4352-b713-ecbb082847f6\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.709630 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.709583 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/17f6623a-31b9-4352-b713-ecbb082847f6-manifests\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-xvsph\" (UID: \"17f6623a-31b9-4352-b713-ecbb082847f6\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.711228 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.711209 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17f6623a-31b9-4352-b713-ecbb082847f6-tmp\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-xvsph\" (UID: \"17f6623a-31b9-4352-b713-ecbb082847f6\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.711320 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.711233 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17f6623a-31b9-4352-b713-ecbb082847f6-webhook-cert\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-xvsph\" (UID: \"17f6623a-31b9-4352-b713-ecbb082847f6\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.711320 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.711256 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/17f6623a-31b9-4352-b713-ecbb082847f6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-xvsph\" (UID: \"17f6623a-31b9-4352-b713-ecbb082847f6\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.723093 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.723064 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znztk\" (UniqueName: \"kubernetes.io/projected/17f6623a-31b9-4352-b713-ecbb082847f6-kube-api-access-znztk\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-xvsph\" (UID: \"17f6623a-31b9-4352-b713-ecbb082847f6\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:43.891600 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:43.891577 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:44.166281 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:44.166259 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph"] Sep 08 10:06:44.168787 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:06:44.168763 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f6623a_31b9_4352_b713_ecbb082847f6.slice/crio-bd3ff4fd9aa0a2e11e249cd2f0a6b6cb11141bbafd60180ac96db6ad46e7344d WatchSource:0}: Error finding container bd3ff4fd9aa0a2e11e249cd2f0a6b6cb11141bbafd60180ac96db6ad46e7344d: Status 404 returned error can't find the container with id bd3ff4fd9aa0a2e11e249cd2f0a6b6cb11141bbafd60180ac96db6ad46e7344d Sep 08 10:06:44.967176 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:44.967141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" event={"ID":"17f6623a-31b9-4352-b713-ecbb082847f6","Type":"ContainerStarted","Data":"bd3ff4fd9aa0a2e11e249cd2f0a6b6cb11141bbafd60180ac96db6ad46e7344d"} Sep 08 10:06:45.946075 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:45.946038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:45.973872 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:45.973839 2576 generic.go:356] "Generic (PLEG): container finished" podID="7316bd8c-b4e8-4642-9722-8b44c7744e67" containerID="7f6ea723069b78a87a795d2753ee5b9a61019714a30b7ad9e52f2497f1c6bb69" exitCode=0 Sep 08 10:06:45.974292 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:45.973921 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" event={"ID":"7316bd8c-b4e8-4642-9722-8b44c7744e67","Type":"ContainerDied","Data":"7f6ea723069b78a87a795d2753ee5b9a61019714a30b7ad9e52f2497f1c6bb69"} Sep 08 10:06:46.009889 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:46.009852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:46.981235 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:46.981205 2576 generic.go:356] "Generic (PLEG): container finished" podID="7316bd8c-b4e8-4642-9722-8b44c7744e67" containerID="299c675833e3933caffd23717da54dd63b7ce8f889814c93fe1bb4f4ccfbb2d0" exitCode=0 Sep 08 10:06:46.981587 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:46.981255 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" event={"ID":"7316bd8c-b4e8-4642-9722-8b44c7744e67","Type":"ContainerDied","Data":"299c675833e3933caffd23717da54dd63b7ce8f889814c93fe1bb4f4ccfbb2d0"} Sep 08 10:06:47.826548 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:47.826488 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:47.980142 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:47.980096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:48.117397 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:48.117373 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" Sep 08 10:06:48.150128 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:48.150099 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7316bd8c-b4e8-4642-9722-8b44c7744e67-bundle\") pod \"7316bd8c-b4e8-4642-9722-8b44c7744e67\" (UID: \"7316bd8c-b4e8-4642-9722-8b44c7744e67\") " Sep 08 10:06:48.150266 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:48.150151 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s8zf\" (UniqueName: \"kubernetes.io/projected/7316bd8c-b4e8-4642-9722-8b44c7744e67-kube-api-access-8s8zf\") pod \"7316bd8c-b4e8-4642-9722-8b44c7744e67\" (UID: \"7316bd8c-b4e8-4642-9722-8b44c7744e67\") " Sep 08 10:06:48.150266 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:48.150211 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7316bd8c-b4e8-4642-9722-8b44c7744e67-util\") pod \"7316bd8c-b4e8-4642-9722-8b44c7744e67\" (UID: \"7316bd8c-b4e8-4642-9722-8b44c7744e67\") " Sep 08 10:06:48.151059 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:48.151029 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7316bd8c-b4e8-4642-9722-8b44c7744e67-bundle" (OuterVolumeSpecName: "bundle") pod "7316bd8c-b4e8-4642-9722-8b44c7744e67" (UID: "7316bd8c-b4e8-4642-9722-8b44c7744e67"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:06:48.152564 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:48.152529 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7316bd8c-b4e8-4642-9722-8b44c7744e67-kube-api-access-8s8zf" (OuterVolumeSpecName: "kube-api-access-8s8zf") pod "7316bd8c-b4e8-4642-9722-8b44c7744e67" (UID: "7316bd8c-b4e8-4642-9722-8b44c7744e67"). InnerVolumeSpecName "kube-api-access-8s8zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:06:48.154837 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:48.154815 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7316bd8c-b4e8-4642-9722-8b44c7744e67-util" (OuterVolumeSpecName: "util") pod "7316bd8c-b4e8-4642-9722-8b44c7744e67" (UID: "7316bd8c-b4e8-4642-9722-8b44c7744e67"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:06:48.251380 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:48.251342 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s8zf\" (UniqueName: \"kubernetes.io/projected/7316bd8c-b4e8-4642-9722-8b44c7744e67-kube-api-access-8s8zf\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:06:48.251529 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:48.251403 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7316bd8c-b4e8-4642-9722-8b44c7744e67-util\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:06:48.251529 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:48.251427 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7316bd8c-b4e8-4642-9722-8b44c7744e67-bundle\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:06:48.992841 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:48.992749 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" event={"ID":"17f6623a-31b9-4352-b713-ecbb082847f6","Type":"ContainerStarted","Data":"98c26dcfba707c8281a9c11052f25e5db3db5bc849d1cd48f0aa302e70338e6f"} Sep 08 10:06:48.994800 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:48.994780 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" Sep 08 10:06:48.994904 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:48.994783 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c75sx" event={"ID":"7316bd8c-b4e8-4642-9722-8b44c7744e67","Type":"ContainerDied","Data":"78328bc5f226aebee6879781bdab59aaa86f1cc9163a106a2bdb21678b739a5a"} Sep 08 10:06:48.994904 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:48.994879 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78328bc5f226aebee6879781bdab59aaa86f1cc9163a106a2bdb21678b739a5a" Sep 08 10:06:49.733006 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:49.732970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:50.000448 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:50.000365 2576 generic.go:356] "Generic (PLEG): container finished" podID="17f6623a-31b9-4352-b713-ecbb082847f6" containerID="98c26dcfba707c8281a9c11052f25e5db3db5bc849d1cd48f0aa302e70338e6f" exitCode=0 Sep 08 10:06:50.000448 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:50.000438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" event={"ID":"17f6623a-31b9-4352-b713-ecbb082847f6","Type":"ContainerDied","Data":"98c26dcfba707c8281a9c11052f25e5db3db5bc849d1cd48f0aa302e70338e6f"} Sep 08 10:06:51.008886 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:51.008851 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" event={"ID":"17f6623a-31b9-4352-b713-ecbb082847f6","Type":"ContainerStarted","Data":"513672b4b0b74ad7ccc57987c3654967f52944ffb2702730955200bcb9bec876"} Sep 08 10:06:51.009280 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:51.008934 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:06:51.029784 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:51.029742 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" podStartSLOduration=2.76334452 podStartE2EDuration="8.029728229s" totalImagesPullingTime="4.191094513s" totalInitContainerRuntime="1.075289196s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:06:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:06:51.028854076 +0000 UTC m=+320.817905879" watchObservedRunningTime="2026-09-08 10:06:51.029728229 +0000 UTC m=+320.818780011" Sep 08 10:06:51.761765 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:51.761732 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:51.862077 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:51.862044 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:53.957386 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:53.957353 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:06:54.057459 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:06:54.057426 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:02.016553 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:02.016524 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-xvsph" Sep 08 10:07:04.820054 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:04.820008 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:04.946312 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:04.946272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:05.130461 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.130372 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:05.135726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.135696 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr"] Sep 08 10:07:05.135978 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.135965 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7316bd8c-b4e8-4642-9722-8b44c7744e67" containerName="extract" Sep 08 10:07:05.136022 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.135980 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7316bd8c-b4e8-4642-9722-8b44c7744e67" containerName="extract" Sep 08 10:07:05.136022 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.135995 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7316bd8c-b4e8-4642-9722-8b44c7744e67" containerName="pull" Sep 08 10:07:05.136022 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.136001 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7316bd8c-b4e8-4642-9722-8b44c7744e67" containerName="pull" Sep 08 10:07:05.136022 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.136010 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7316bd8c-b4e8-4642-9722-8b44c7744e67" containerName="util" Sep 08 10:07:05.136022 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.136015 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7316bd8c-b4e8-4642-9722-8b44c7744e67" containerName="util" Sep 08 10:07:05.136170 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.136069 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7316bd8c-b4e8-4642-9722-8b44c7744e67" containerName="extract" Sep 08 10:07:05.145232 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.145203 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" Sep 08 10:07:05.148575 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.148554 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 08 10:07:05.148703 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.148656 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 08 10:07:05.148797 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.148780 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 08 10:07:05.149164 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.149147 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 08 10:07:05.155642 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.155604 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr"] Sep 08 10:07:05.285106 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.285072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p6kn\" (UniqueName: \"kubernetes.io/projected/53096f12-27bc-4599-8c5c-a31c72bc3eee-kube-api-access-7p6kn\") pod \"kube-auth-proxy-77c9fc4cdc-v9jpr\" (UID: \"53096f12-27bc-4599-8c5c-a31c72bc3eee\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" Sep 08 10:07:05.285270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.285122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53096f12-27bc-4599-8c5c-a31c72bc3eee-tls-certs\") pod \"kube-auth-proxy-77c9fc4cdc-v9jpr\" (UID: \"53096f12-27bc-4599-8c5c-a31c72bc3eee\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" Sep 08 10:07:05.285270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.285140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53096f12-27bc-4599-8c5c-a31c72bc3eee-tmp\") pod \"kube-auth-proxy-77c9fc4cdc-v9jpr\" (UID: \"53096f12-27bc-4599-8c5c-a31c72bc3eee\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" Sep 08 10:07:05.386450 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.386365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p6kn\" (UniqueName: \"kubernetes.io/projected/53096f12-27bc-4599-8c5c-a31c72bc3eee-kube-api-access-7p6kn\") pod \"kube-auth-proxy-77c9fc4cdc-v9jpr\" (UID: \"53096f12-27bc-4599-8c5c-a31c72bc3eee\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" Sep 08 10:07:05.386450 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.386416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53096f12-27bc-4599-8c5c-a31c72bc3eee-tls-certs\") pod \"kube-auth-proxy-77c9fc4cdc-v9jpr\" (UID: \"53096f12-27bc-4599-8c5c-a31c72bc3eee\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" Sep 08 10:07:05.386450 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.386435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53096f12-27bc-4599-8c5c-a31c72bc3eee-tmp\") pod \"kube-auth-proxy-77c9fc4cdc-v9jpr\" (UID: \"53096f12-27bc-4599-8c5c-a31c72bc3eee\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" Sep 08 10:07:05.386675 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:07:05.386537 2576 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 08 10:07:05.386675 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:07:05.386646 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53096f12-27bc-4599-8c5c-a31c72bc3eee-tls-certs podName:53096f12-27bc-4599-8c5c-a31c72bc3eee nodeName:}" failed. No retries permitted until 2026-09-08 10:07:05.886599725 +0000 UTC m=+335.675651514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/53096f12-27bc-4599-8c5c-a31c72bc3eee-tls-certs") pod "kube-auth-proxy-77c9fc4cdc-v9jpr" (UID: "53096f12-27bc-4599-8c5c-a31c72bc3eee") : secret "kube-auth-proxy-tls" not found Sep 08 10:07:05.388702 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.388668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53096f12-27bc-4599-8c5c-a31c72bc3eee-tmp\") pod \"kube-auth-proxy-77c9fc4cdc-v9jpr\" (UID: \"53096f12-27bc-4599-8c5c-a31c72bc3eee\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" Sep 08 10:07:05.405306 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.405283 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p6kn\" (UniqueName: \"kubernetes.io/projected/53096f12-27bc-4599-8c5c-a31c72bc3eee-kube-api-access-7p6kn\") pod \"kube-auth-proxy-77c9fc4cdc-v9jpr\" (UID: \"53096f12-27bc-4599-8c5c-a31c72bc3eee\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" Sep 08 10:07:05.891047 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.891011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53096f12-27bc-4599-8c5c-a31c72bc3eee-tls-certs\") pod \"kube-auth-proxy-77c9fc4cdc-v9jpr\" (UID: \"53096f12-27bc-4599-8c5c-a31c72bc3eee\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" Sep 08 10:07:05.891503 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:07:05.891155 2576 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 08 10:07:05.891503 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:07:05.891209 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53096f12-27bc-4599-8c5c-a31c72bc3eee-tls-certs podName:53096f12-27bc-4599-8c5c-a31c72bc3eee nodeName:}" failed. No retries permitted until 2026-09-08 10:07:06.89119587 +0000 UTC m=+336.680247651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/53096f12-27bc-4599-8c5c-a31c72bc3eee-tls-certs") pod "kube-auth-proxy-77c9fc4cdc-v9jpr" (UID: "53096f12-27bc-4599-8c5c-a31c72bc3eee") : secret "kube-auth-proxy-tls" not found Sep 08 10:07:05.953896 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:05.953862 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:06.821404 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.821367 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt"] Sep 08 10:07:06.823528 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.823512 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:06.827157 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.827128 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:06.828705 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.828684 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 10:07:06.828857 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.828842 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 08 10:07:06.828998 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.828977 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 08 10:07:06.829067 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.829055 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 08 10:07:06.829445 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.829432 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xhchn\"" Sep 08 10:07:06.844382 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.844357 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt"] Sep 08 10:07:06.899291 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.899251 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:06.899291 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.899293 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtp59\" (UniqueName: \"kubernetes.io/projected/fe5e2253-d095-465d-adc3-e4b92e39e34c-kube-api-access-xtp59\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:06.899796 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.899325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:06.899796 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.899348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe5e2253-d095-465d-adc3-e4b92e39e34c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:06.899796 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.899374 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:06.899796 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.899452 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:06.899796 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.899501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:06.899796 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.899533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53096f12-27bc-4599-8c5c-a31c72bc3eee-tls-certs\") pod \"kube-auth-proxy-77c9fc4cdc-v9jpr\" (UID: \"53096f12-27bc-4599-8c5c-a31c72bc3eee\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" Sep 08 10:07:06.901972 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.901953 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53096f12-27bc-4599-8c5c-a31c72bc3eee-tls-certs\") pod \"kube-auth-proxy-77c9fc4cdc-v9jpr\" (UID: \"53096f12-27bc-4599-8c5c-a31c72bc3eee\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" Sep 08 10:07:06.922811 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.922764 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:06.955649 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:06.955610 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" Sep 08 10:07:07.000570 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.000342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:07.000570 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.000430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:07.000570 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.000462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtp59\" (UniqueName: \"kubernetes.io/projected/fe5e2253-d095-465d-adc3-e4b92e39e34c-kube-api-access-xtp59\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:07.000570 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.000498 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:07.000570 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.000528 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe5e2253-d095-465d-adc3-e4b92e39e34c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:07.000570 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.000554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:07.000938 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.000591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:07.002129 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.002054 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:07.004063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.004022 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:07.004497 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.004471 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:07.006416 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.006389 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe5e2253-d095-465d-adc3-e4b92e39e34c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:07.007182 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.007155 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:07.018646 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.018594 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtp59\" (UniqueName: \"kubernetes.io/projected/fe5e2253-d095-465d-adc3-e4b92e39e34c-kube-api-access-xtp59\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:07.019444 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.018871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cqrqt\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:07.132991 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.132908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:07.227199 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.227171 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr"] Sep 08 10:07:07.229039 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:07:07.229013 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53096f12_27bc_4599_8c5c_a31c72bc3eee.slice/crio-8c86f1bb1b6f12f1dde60357d1a8661108c06abd8705a1d7fe75e0d229212e7f WatchSource:0}: Error finding container 8c86f1bb1b6f12f1dde60357d1a8661108c06abd8705a1d7fe75e0d229212e7f: Status 404 returned error can't find the container with id 8c86f1bb1b6f12f1dde60357d1a8661108c06abd8705a1d7fe75e0d229212e7f Sep 08 10:07:07.407725 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.407697 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt"] Sep 08 10:07:07.410205 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:07:07.410181 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5e2253_d095_465d_adc3_e4b92e39e34c.slice/crio-1dcd391be8dc9e56eed7245aa5a3b88d00e734bc17739d31507ec8b0cdd10a03 WatchSource:0}: Error finding container 1dcd391be8dc9e56eed7245aa5a3b88d00e734bc17739d31507ec8b0cdd10a03: Status 404 returned error can't find the container with id 1dcd391be8dc9e56eed7245aa5a3b88d00e734bc17739d31507ec8b0cdd10a03 Sep 08 10:07:07.901511 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:07.901478 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:08.083379 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:08.083344 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" event={"ID":"fe5e2253-d095-465d-adc3-e4b92e39e34c","Type":"ContainerStarted","Data":"1dcd391be8dc9e56eed7245aa5a3b88d00e734bc17739d31507ec8b0cdd10a03"} Sep 08 10:07:08.084740 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:08.084708 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" event={"ID":"53096f12-27bc-4599-8c5c-a31c72bc3eee","Type":"ContainerStarted","Data":"8c86f1bb1b6f12f1dde60357d1a8661108c06abd8705a1d7fe75e0d229212e7f"} Sep 08 10:07:12.108523 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:12.108468 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" event={"ID":"53096f12-27bc-4599-8c5c-a31c72bc3eee","Type":"ContainerStarted","Data":"f5d24f6ecbeb8f5bf1965c9c43288ec5cb416405afa01bbe24c32783a591c8ed"} Sep 08 10:07:12.125341 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:12.125278 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-v9jpr" podStartSLOduration=3.143078076 podStartE2EDuration="7.125255057s" totalImagesPullingTime="3.982176981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:07:12.123765525 +0000 UTC m=+341.912817329" watchObservedRunningTime="2026-09-08 10:07:12.125255057 +0000 UTC m=+341.914306862" Sep 08 10:07:13.713401 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:13.713333 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:07:13.713678 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:13.713435 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:07:14.121282 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:14.121185 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" event={"ID":"fe5e2253-d095-465d-adc3-e4b92e39e34c","Type":"ContainerStarted","Data":"a890768c56b8b20ad90d891b7c876eff3699fa5880c53ce82243eef1fb96a5b8"} Sep 08 10:07:14.121511 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:14.121381 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:14.122949 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:14.122921 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cqrqt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 10:07:14.123071 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:14.122974 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" podUID="fe5e2253-d095-465d-adc3-e4b92e39e34c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 10:07:14.145906 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:14.145860 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" podStartSLOduration=1.844893721 podStartE2EDuration="8.145846272s" totalImagesPullingTime="6.300952551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:07:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:07:14.144408671 +0000 UTC m=+343.933460476" watchObservedRunningTime="2026-09-08 10:07:14.145846272 +0000 UTC m=+343.934898076" Sep 08 10:07:14.790516 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:14.790477 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:14.866404 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:14.866367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:15.126372 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:15.126290 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cqrqt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 10:07:15.126372 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:15.126352 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" podUID="fe5e2253-d095-465d-adc3-e4b92e39e34c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 10:07:15.556240 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:15.556203 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:15.984750 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:15.984713 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n9qmd"] Sep 08 10:07:15.987804 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:15.987787 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-n9qmd" Sep 08 10:07:15.991129 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:15.991105 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 08 10:07:15.991675 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:15.991653 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-z4ncl\"" Sep 08 10:07:15.998925 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:15.998903 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n9qmd"] Sep 08 10:07:16.080188 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.080156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e92279a-8448-4bcf-9b75-4b80f73c3f9d-cert\") pod \"odh-model-controller-7998c67dc4-n9qmd\" (UID: \"2e92279a-8448-4bcf-9b75-4b80f73c3f9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9qmd" Sep 08 10:07:16.080319 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.080218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn2d4\" (UniqueName: \"kubernetes.io/projected/2e92279a-8448-4bcf-9b75-4b80f73c3f9d-kube-api-access-xn2d4\") pod \"odh-model-controller-7998c67dc4-n9qmd\" (UID: \"2e92279a-8448-4bcf-9b75-4b80f73c3f9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9qmd" Sep 08 10:07:16.181263 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.181217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e92279a-8448-4bcf-9b75-4b80f73c3f9d-cert\") pod \"odh-model-controller-7998c67dc4-n9qmd\" (UID: \"2e92279a-8448-4bcf-9b75-4b80f73c3f9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9qmd" Sep 08 10:07:16.181433 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.181303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn2d4\" (UniqueName: \"kubernetes.io/projected/2e92279a-8448-4bcf-9b75-4b80f73c3f9d-kube-api-access-xn2d4\") pod \"odh-model-controller-7998c67dc4-n9qmd\" (UID: \"2e92279a-8448-4bcf-9b75-4b80f73c3f9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9qmd" Sep 08 10:07:16.181433 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:07:16.181378 2576 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 08 10:07:16.181501 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:07:16.181461 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e92279a-8448-4bcf-9b75-4b80f73c3f9d-cert podName:2e92279a-8448-4bcf-9b75-4b80f73c3f9d nodeName:}" failed. No retries permitted until 2026-09-08 10:07:16.681439535 +0000 UTC m=+346.470491317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2e92279a-8448-4bcf-9b75-4b80f73c3f9d-cert") pod "odh-model-controller-7998c67dc4-n9qmd" (UID: "2e92279a-8448-4bcf-9b75-4b80f73c3f9d") : secret "odh-model-controller-webhook-cert" not found Sep 08 10:07:16.197648 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.197603 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn2d4\" (UniqueName: \"kubernetes.io/projected/2e92279a-8448-4bcf-9b75-4b80f73c3f9d-kube-api-access-xn2d4\") pod \"odh-model-controller-7998c67dc4-n9qmd\" (UID: \"2e92279a-8448-4bcf-9b75-4b80f73c3f9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9qmd" Sep 08 10:07:16.532802 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.532772 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-84rwh"] Sep 08 10:07:16.535905 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.535888 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-84rwh" Sep 08 10:07:16.538600 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.538579 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 08 10:07:16.551931 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.551903 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-84rwh"] Sep 08 10:07:16.685423 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.685386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgqh6\" (UniqueName: \"kubernetes.io/projected/563497e2-7695-4b69-89a8-e9f44721c58f-kube-api-access-fgqh6\") pod \"model-serving-api-54d89bd5fc-84rwh\" (UID: \"563497e2-7695-4b69-89a8-e9f44721c58f\") " pod="opendatahub/model-serving-api-54d89bd5fc-84rwh" Sep 08 10:07:16.685605 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.685454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563497e2-7695-4b69-89a8-e9f44721c58f-tls-certs\") pod \"model-serving-api-54d89bd5fc-84rwh\" (UID: \"563497e2-7695-4b69-89a8-e9f44721c58f\") " pod="opendatahub/model-serving-api-54d89bd5fc-84rwh" Sep 08 10:07:16.685605 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.685524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e92279a-8448-4bcf-9b75-4b80f73c3f9d-cert\") pod \"odh-model-controller-7998c67dc4-n9qmd\" (UID: \"2e92279a-8448-4bcf-9b75-4b80f73c3f9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9qmd" Sep 08 10:07:16.687899 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.687875 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e92279a-8448-4bcf-9b75-4b80f73c3f9d-cert\") pod \"odh-model-controller-7998c67dc4-n9qmd\" (UID: \"2e92279a-8448-4bcf-9b75-4b80f73c3f9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9qmd" Sep 08 10:07:16.786205 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.786116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgqh6\" (UniqueName: \"kubernetes.io/projected/563497e2-7695-4b69-89a8-e9f44721c58f-kube-api-access-fgqh6\") pod \"model-serving-api-54d89bd5fc-84rwh\" (UID: \"563497e2-7695-4b69-89a8-e9f44721c58f\") " pod="opendatahub/model-serving-api-54d89bd5fc-84rwh" Sep 08 10:07:16.786205 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.786168 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563497e2-7695-4b69-89a8-e9f44721c58f-tls-certs\") pod \"model-serving-api-54d89bd5fc-84rwh\" (UID: \"563497e2-7695-4b69-89a8-e9f44721c58f\") " pod="opendatahub/model-serving-api-54d89bd5fc-84rwh" Sep 08 10:07:16.786380 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:07:16.786264 2576 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 08 10:07:16.786380 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:07:16.786327 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/563497e2-7695-4b69-89a8-e9f44721c58f-tls-certs podName:563497e2-7695-4b69-89a8-e9f44721c58f nodeName:}" failed. No retries permitted until 2026-09-08 10:07:17.286307435 +0000 UTC m=+347.075359223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/563497e2-7695-4b69-89a8-e9f44721c58f-tls-certs") pod "model-serving-api-54d89bd5fc-84rwh" (UID: "563497e2-7695-4b69-89a8-e9f44721c58f") : secret "model-serving-api-tls" not found Sep 08 10:07:16.796500 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.796470 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgqh6\" (UniqueName: \"kubernetes.io/projected/563497e2-7695-4b69-89a8-e9f44721c58f-kube-api-access-fgqh6\") pod \"model-serving-api-54d89bd5fc-84rwh\" (UID: \"563497e2-7695-4b69-89a8-e9f44721c58f\") " pod="opendatahub/model-serving-api-54d89bd5fc-84rwh" Sep 08 10:07:16.897482 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.897446 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-n9qmd" Sep 08 10:07:16.907832 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.907684 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:16.965106 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:16.965069 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:17.173553 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:17.173509 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n9qmd"] Sep 08 10:07:17.175684 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:07:17.175655 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e92279a_8448_4bcf_9b75_4b80f73c3f9d.slice/crio-a12890b5f24af548a1e3f33d7d224b1a8e5d92e57f50ea104a54c08abba13e7f WatchSource:0}: Error finding container a12890b5f24af548a1e3f33d7d224b1a8e5d92e57f50ea104a54c08abba13e7f: Status 404 returned error can't find the container with id a12890b5f24af548a1e3f33d7d224b1a8e5d92e57f50ea104a54c08abba13e7f Sep 08 10:07:17.290024 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:17.289990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563497e2-7695-4b69-89a8-e9f44721c58f-tls-certs\") pod \"model-serving-api-54d89bd5fc-84rwh\" (UID: \"563497e2-7695-4b69-89a8-e9f44721c58f\") " pod="opendatahub/model-serving-api-54d89bd5fc-84rwh" Sep 08 10:07:17.292542 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:17.292507 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563497e2-7695-4b69-89a8-e9f44721c58f-tls-certs\") pod \"model-serving-api-54d89bd5fc-84rwh\" (UID: \"563497e2-7695-4b69-89a8-e9f44721c58f\") " pod="opendatahub/model-serving-api-54d89bd5fc-84rwh" Sep 08 10:07:17.445516 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:17.445416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-84rwh" Sep 08 10:07:17.714142 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:17.714119 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-84rwh"] Sep 08 10:07:17.716283 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:07:17.716252 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563497e2_7695_4b69_89a8_e9f44721c58f.slice/crio-df577067ca9f62a5a94b8b480095b01ce8dcbcd94a3b9becbdfdb545f9fd9ea0 WatchSource:0}: Error finding container df577067ca9f62a5a94b8b480095b01ce8dcbcd94a3b9becbdfdb545f9fd9ea0: Status 404 returned error can't find the container with id df577067ca9f62a5a94b8b480095b01ce8dcbcd94a3b9becbdfdb545f9fd9ea0 Sep 08 10:07:18.127086 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:18.127012 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:18.141715 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:18.141684 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-84rwh" event={"ID":"563497e2-7695-4b69-89a8-e9f44721c58f","Type":"ContainerStarted","Data":"df577067ca9f62a5a94b8b480095b01ce8dcbcd94a3b9becbdfdb545f9fd9ea0"} Sep 08 10:07:18.142856 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:18.142833 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-n9qmd" event={"ID":"2e92279a-8448-4bcf-9b75-4b80f73c3f9d","Type":"ContainerStarted","Data":"a12890b5f24af548a1e3f33d7d224b1a8e5d92e57f50ea104a54c08abba13e7f"} Sep 08 10:07:20.159669 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:20.159612 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-84rwh" event={"ID":"563497e2-7695-4b69-89a8-e9f44721c58f","Type":"ContainerStarted","Data":"8a75fc1b1db0fc5304a7aaa2e95f87626b933966d68f960c7c06c0b81e5c1685"} Sep 08 10:07:20.160091 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:20.159707 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-84rwh" Sep 08 10:07:20.176323 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:20.176278 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-84rwh" podStartSLOduration=2.570905209 podStartE2EDuration="4.176263253s" totalImagesPullingTime="1.605358044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:07:20.173743654 +0000 UTC m=+349.962795459" watchObservedRunningTime="2026-09-08 10:07:20.176263253 +0000 UTC m=+349.965315079" Sep 08 10:07:21.168163 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:21.168118 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-n9qmd" event={"ID":"2e92279a-8448-4bcf-9b75-4b80f73c3f9d","Type":"ContainerStarted","Data":"be8af474bfee67662ca2283cdd40cae6bb55a8e57c7215106518782fe77f03dd"} Sep 08 10:07:21.168644 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:21.168425 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-n9qmd" Sep 08 10:07:21.185302 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:21.185250 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-n9qmd" podStartSLOduration=2.6784676469999997 podStartE2EDuration="6.185229798s" totalImagesPullingTime="3.506762151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:07:21.184067009 +0000 UTC m=+350.973118813" watchObservedRunningTime="2026-09-08 10:07:21.185229798 +0000 UTC m=+350.974281602" Sep 08 10:07:26.299471 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.299387 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-25nft"] Sep 08 10:07:26.302685 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.302659 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-25nft" Sep 08 10:07:26.306690 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.306604 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 08 10:07:26.306690 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.306666 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vfs8n\"" Sep 08 10:07:26.323017 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.322991 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r"] Sep 08 10:07:26.334859 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.334840 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-25nft"] Sep 08 10:07:26.334972 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.334955 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r" Sep 08 10:07:26.338182 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.338164 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 08 10:07:26.338286 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.338209 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-ng47m\"" Sep 08 10:07:26.353173 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.353143 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r"] Sep 08 10:07:26.473027 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.472990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4v8p\" (UniqueName: \"kubernetes.io/projected/250f37c3-df6f-44b0-b16c-6a14c05d3bd8-kube-api-access-m4v8p\") pod \"kserve-controller-manager-d6f4b65b8-25nft\" (UID: \"250f37c3-df6f-44b0-b16c-6a14c05d3bd8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-25nft" Sep 08 10:07:26.473027 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.473030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/250f37c3-df6f-44b0-b16c-6a14c05d3bd8-cert\") pod \"kserve-controller-manager-d6f4b65b8-25nft\" (UID: \"250f37c3-df6f-44b0-b16c-6a14c05d3bd8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-25nft" Sep 08 10:07:26.473236 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.473054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk6g7\" (UniqueName: \"kubernetes.io/projected/1ab54ac7-8c16-4fbd-b68f-02884684490e-kube-api-access-hk6g7\") pod \"llmisvc-controller-manager-777c6bd7d5-wch2r\" (UID: \"1ab54ac7-8c16-4fbd-b68f-02884684490e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r" Sep 08 10:07:26.473236 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.473155 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ab54ac7-8c16-4fbd-b68f-02884684490e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wch2r\" (UID: \"1ab54ac7-8c16-4fbd-b68f-02884684490e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r" Sep 08 10:07:26.573924 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.573827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4v8p\" (UniqueName: \"kubernetes.io/projected/250f37c3-df6f-44b0-b16c-6a14c05d3bd8-kube-api-access-m4v8p\") pod \"kserve-controller-manager-d6f4b65b8-25nft\" (UID: \"250f37c3-df6f-44b0-b16c-6a14c05d3bd8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-25nft" Sep 08 10:07:26.573924 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.573883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/250f37c3-df6f-44b0-b16c-6a14c05d3bd8-cert\") pod \"kserve-controller-manager-d6f4b65b8-25nft\" (UID: \"250f37c3-df6f-44b0-b16c-6a14c05d3bd8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-25nft" Sep 08 10:07:26.573924 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.573919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk6g7\" (UniqueName: \"kubernetes.io/projected/1ab54ac7-8c16-4fbd-b68f-02884684490e-kube-api-access-hk6g7\") pod \"llmisvc-controller-manager-777c6bd7d5-wch2r\" (UID: \"1ab54ac7-8c16-4fbd-b68f-02884684490e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r" Sep 08 10:07:26.574184 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:07:26.574049 2576 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 08 10:07:26.574184 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.574074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ab54ac7-8c16-4fbd-b68f-02884684490e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wch2r\" (UID: \"1ab54ac7-8c16-4fbd-b68f-02884684490e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r" Sep 08 10:07:26.574184 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:07:26.574120 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/250f37c3-df6f-44b0-b16c-6a14c05d3bd8-cert podName:250f37c3-df6f-44b0-b16c-6a14c05d3bd8 nodeName:}" failed. No retries permitted until 2026-09-08 10:07:27.074097266 +0000 UTC m=+356.863149057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/250f37c3-df6f-44b0-b16c-6a14c05d3bd8-cert") pod "kserve-controller-manager-d6f4b65b8-25nft" (UID: "250f37c3-df6f-44b0-b16c-6a14c05d3bd8") : secret "kserve-webhook-server-cert" not found Sep 08 10:07:26.576556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.576525 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ab54ac7-8c16-4fbd-b68f-02884684490e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wch2r\" (UID: \"1ab54ac7-8c16-4fbd-b68f-02884684490e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r" Sep 08 10:07:26.591037 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.591008 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4v8p\" (UniqueName: \"kubernetes.io/projected/250f37c3-df6f-44b0-b16c-6a14c05d3bd8-kube-api-access-m4v8p\") pod \"kserve-controller-manager-d6f4b65b8-25nft\" (UID: \"250f37c3-df6f-44b0-b16c-6a14c05d3bd8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-25nft" Sep 08 10:07:26.615324 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.615299 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk6g7\" (UniqueName: \"kubernetes.io/projected/1ab54ac7-8c16-4fbd-b68f-02884684490e-kube-api-access-hk6g7\") pod \"llmisvc-controller-manager-777c6bd7d5-wch2r\" (UID: \"1ab54ac7-8c16-4fbd-b68f-02884684490e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r" Sep 08 10:07:26.645153 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.645126 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r" Sep 08 10:07:26.925273 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:26.925211 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r"] Sep 08 10:07:26.927201 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:07:26.927173 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1ab54ac7_8c16_4fbd_b68f_02884684490e.slice/crio-7c8ce6a11c4739759cc844ba8d66022976838b7a56e42930620582db068ddb81 WatchSource:0}: Error finding container 7c8ce6a11c4739759cc844ba8d66022976838b7a56e42930620582db068ddb81: Status 404 returned error can't find the container with id 7c8ce6a11c4739759cc844ba8d66022976838b7a56e42930620582db068ddb81 Sep 08 10:07:27.077368 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:27.077336 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/250f37c3-df6f-44b0-b16c-6a14c05d3bd8-cert\") pod \"kserve-controller-manager-d6f4b65b8-25nft\" (UID: \"250f37c3-df6f-44b0-b16c-6a14c05d3bd8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-25nft" Sep 08 10:07:27.079737 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:27.079712 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/250f37c3-df6f-44b0-b16c-6a14c05d3bd8-cert\") pod \"kserve-controller-manager-d6f4b65b8-25nft\" (UID: \"250f37c3-df6f-44b0-b16c-6a14c05d3bd8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-25nft" Sep 08 10:07:27.195911 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:27.195826 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r" event={"ID":"1ab54ac7-8c16-4fbd-b68f-02884684490e","Type":"ContainerStarted","Data":"7c8ce6a11c4739759cc844ba8d66022976838b7a56e42930620582db068ddb81"} Sep 08 10:07:27.213056 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:27.213033 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-25nft" Sep 08 10:07:27.513807 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:27.513779 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-25nft"] Sep 08 10:07:28.108446 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.108408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:28.201384 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.201349 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-25nft" event={"ID":"250f37c3-df6f-44b0-b16c-6a14c05d3bd8","Type":"ContainerStarted","Data":"faf5e5fda9da67037de3ecffd8a74a335f8729e23582eb4d2aee7479161bba16"} Sep 08 10:07:28.335646 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.335600 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt"] Sep 08 10:07:28.335819 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.335798 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" podUID="fe5e2253-d095-465d-adc3-e4b92e39e34c" containerName="discovery" containerID="cri-o://a890768c56b8b20ad90d891b7c876eff3699fa5880c53ce82243eef1fb96a5b8" gracePeriod=30 Sep 08 10:07:28.603170 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.603140 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:28.689571 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.689542 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe5e2253-d095-465d-adc3-e4b92e39e34c-local-certs\") pod \"fe5e2253-d095-465d-adc3-e4b92e39e34c\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " Sep 08 10:07:28.689775 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.689581 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-token\") pod \"fe5e2253-d095-465d-adc3-e4b92e39e34c\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " Sep 08 10:07:28.689775 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.689604 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-csr-dns-cert\") pod \"fe5e2253-d095-465d-adc3-e4b92e39e34c\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " Sep 08 10:07:28.689775 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.689644 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-csr-ca-configmap\") pod \"fe5e2253-d095-465d-adc3-e4b92e39e34c\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " Sep 08 10:07:28.690083 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.690050 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtp59\" (UniqueName: \"kubernetes.io/projected/fe5e2253-d095-465d-adc3-e4b92e39e34c-kube-api-access-xtp59\") pod \"fe5e2253-d095-465d-adc3-e4b92e39e34c\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " Sep 08 10:07:28.690083 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.690065 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "fe5e2253-d095-465d-adc3-e4b92e39e34c" (UID: "fe5e2253-d095-465d-adc3-e4b92e39e34c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:07:28.690263 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.690104 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-kubeconfig\") pod \"fe5e2253-d095-465d-adc3-e4b92e39e34c\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " Sep 08 10:07:28.690263 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.690168 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-cacerts\") pod \"fe5e2253-d095-465d-adc3-e4b92e39e34c\" (UID: \"fe5e2253-d095-465d-adc3-e4b92e39e34c\") " Sep 08 10:07:28.690728 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.690394 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-csr-ca-configmap\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:07:28.692718 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.692683 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "fe5e2253-d095-465d-adc3-e4b92e39e34c" (UID: "fe5e2253-d095-465d-adc3-e4b92e39e34c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:07:28.692991 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.692959 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-cacerts" (OuterVolumeSpecName: "cacerts") pod "fe5e2253-d095-465d-adc3-e4b92e39e34c" (UID: "fe5e2253-d095-465d-adc3-e4b92e39e34c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:07:28.693501 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.693467 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe5e2253-d095-465d-adc3-e4b92e39e34c" (UID: "fe5e2253-d095-465d-adc3-e4b92e39e34c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:07:28.694092 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.694070 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5e2253-d095-465d-adc3-e4b92e39e34c-kube-api-access-xtp59" (OuterVolumeSpecName: "kube-api-access-xtp59") pod "fe5e2253-d095-465d-adc3-e4b92e39e34c" (UID: "fe5e2253-d095-465d-adc3-e4b92e39e34c"). InnerVolumeSpecName "kube-api-access-xtp59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:07:28.694604 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.694575 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "fe5e2253-d095-465d-adc3-e4b92e39e34c" (UID: "fe5e2253-d095-465d-adc3-e4b92e39e34c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:07:28.694844 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.694821 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5e2253-d095-465d-adc3-e4b92e39e34c-local-certs" (OuterVolumeSpecName: "local-certs") pod "fe5e2253-d095-465d-adc3-e4b92e39e34c" (UID: "fe5e2253-d095-465d-adc3-e4b92e39e34c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:07:28.791315 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.791274 2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe5e2253-d095-465d-adc3-e4b92e39e34c-local-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:07:28.791315 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.791312 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-token\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:07:28.791523 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.791326 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-csr-dns-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:07:28.791523 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.791341 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtp59\" (UniqueName: \"kubernetes.io/projected/fe5e2253-d095-465d-adc3-e4b92e39e34c-kube-api-access-xtp59\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:07:28.791523 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.791356 2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-istio-kubeconfig\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:07:28.791523 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:28.791369 2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe5e2253-d095-465d-adc3-e4b92e39e34c-cacerts\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:07:29.209123 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:29.209085 2576 generic.go:356] "Generic (PLEG): container finished" podID="fe5e2253-d095-465d-adc3-e4b92e39e34c" containerID="a890768c56b8b20ad90d891b7c876eff3699fa5880c53ce82243eef1fb96a5b8" exitCode=0 Sep 08 10:07:29.209332 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:29.209146 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" Sep 08 10:07:29.209332 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:29.209147 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" event={"ID":"fe5e2253-d095-465d-adc3-e4b92e39e34c","Type":"ContainerDied","Data":"a890768c56b8b20ad90d891b7c876eff3699fa5880c53ce82243eef1fb96a5b8"} Sep 08 10:07:29.209332 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:29.209316 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt" event={"ID":"fe5e2253-d095-465d-adc3-e4b92e39e34c","Type":"ContainerDied","Data":"1dcd391be8dc9e56eed7245aa5a3b88d00e734bc17739d31507ec8b0cdd10a03"} Sep 08 10:07:29.209501 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:29.209353 2576 scope.go:117] "RemoveContainer" containerID="a890768c56b8b20ad90d891b7c876eff3699fa5880c53ce82243eef1fb96a5b8" Sep 08 10:07:29.220822 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:29.220797 2576 scope.go:117] "RemoveContainer" containerID="a890768c56b8b20ad90d891b7c876eff3699fa5880c53ce82243eef1fb96a5b8" Sep 08 10:07:29.221149 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:07:29.221126 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a890768c56b8b20ad90d891b7c876eff3699fa5880c53ce82243eef1fb96a5b8\": container with ID starting with a890768c56b8b20ad90d891b7c876eff3699fa5880c53ce82243eef1fb96a5b8 not found: ID does not exist" containerID="a890768c56b8b20ad90d891b7c876eff3699fa5880c53ce82243eef1fb96a5b8" Sep 08 10:07:29.221234 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:29.221160 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a890768c56b8b20ad90d891b7c876eff3699fa5880c53ce82243eef1fb96a5b8"} err="failed to get container status \"a890768c56b8b20ad90d891b7c876eff3699fa5880c53ce82243eef1fb96a5b8\": rpc error: code = NotFound desc = could not find container \"a890768c56b8b20ad90d891b7c876eff3699fa5880c53ce82243eef1fb96a5b8\": container with ID starting with a890768c56b8b20ad90d891b7c876eff3699fa5880c53ce82243eef1fb96a5b8 not found: ID does not exist" Sep 08 10:07:29.236863 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:29.236826 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt"] Sep 08 10:07:29.245831 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:29.245806 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqrqt"] Sep 08 10:07:30.078520 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:30.078484 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:30.677421 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:30.677386 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe5e2253-d095-465d-adc3-e4b92e39e34c" path="/var/lib/kubelet/pods/fe5e2253-d095-465d-adc3-e4b92e39e34c/volumes" Sep 08 10:07:31.172504 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:31.172473 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-84rwh" Sep 08 10:07:31.223509 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:31.223477 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-25nft" event={"ID":"250f37c3-df6f-44b0-b16c-6a14c05d3bd8","Type":"ContainerStarted","Data":"0858b9fcf8ff3b4bf9da5e1c214d5364ffe281189d811de3013af8af5c3d9274"} Sep 08 10:07:31.223713 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:31.223585 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-25nft" Sep 08 10:07:31.225092 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:31.225068 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r" event={"ID":"1ab54ac7-8c16-4fbd-b68f-02884684490e","Type":"ContainerStarted","Data":"d89dd75c149d11ea7e42dee726c1eea9a2c7ee50230aa6c11cf902dc847bee5e"} Sep 08 10:07:31.225204 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:31.225172 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r" Sep 08 10:07:31.239394 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:31.239352 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-25nft" podStartSLOduration=1.878086076 podStartE2EDuration="5.23933736s" totalImagesPullingTime="3.361251284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:07:31.238405857 +0000 UTC m=+361.027457673" watchObservedRunningTime="2026-09-08 10:07:31.23933736 +0000 UTC m=+361.028389164" Sep 08 10:07:31.254683 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:31.254641 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r" podStartSLOduration=1.244370641 podStartE2EDuration="5.254605478s" totalImagesPullingTime="4.010234837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:07:31.253267496 +0000 UTC m=+361.042319299" watchObservedRunningTime="2026-09-08 10:07:31.254605478 +0000 UTC m=+361.043657317" Sep 08 10:07:32.173810 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:32.173777 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-n9qmd" Sep 08 10:07:38.006402 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.006368 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-879476b9f-vc6q4"] Sep 08 10:07:38.006890 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.006721 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5e2253-d095-465d-adc3-e4b92e39e34c" containerName="discovery" Sep 08 10:07:38.006890 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.006735 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5e2253-d095-465d-adc3-e4b92e39e34c" containerName="discovery" Sep 08 10:07:38.006890 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.006796 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5e2253-d095-465d-adc3-e4b92e39e34c" containerName="discovery" Sep 08 10:07:38.009914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.009894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.011986 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.011962 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-brk64\"" Sep 08 10:07:38.012212 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.012193 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 10:07:38.012212 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.012204 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 10:07:38.012904 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.012881 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 10:07:38.013126 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.013107 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 10:07:38.013217 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.013201 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 10:07:38.016948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.016927 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 08 10:07:38.020094 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.020072 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-879476b9f-vc6q4"] Sep 08 10:07:38.056574 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.056551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:38.066450 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.066429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffdc38d6-64a0-4164-88a8-459e2c5ba746-console-oauth-config\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.066541 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.066459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffdc38d6-64a0-4164-88a8-459e2c5ba746-console-config\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.066541 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.066488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffdc38d6-64a0-4164-88a8-459e2c5ba746-oauth-serving-cert\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.066609 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.066569 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffdc38d6-64a0-4164-88a8-459e2c5ba746-service-ca\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.066665 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.066637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl85h\" (UniqueName: \"kubernetes.io/projected/ffdc38d6-64a0-4164-88a8-459e2c5ba746-kube-api-access-dl85h\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.066699 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.066680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffdc38d6-64a0-4164-88a8-459e2c5ba746-console-serving-cert\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.066731 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.066705 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffdc38d6-64a0-4164-88a8-459e2c5ba746-trusted-ca-bundle\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.167501 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.167469 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffdc38d6-64a0-4164-88a8-459e2c5ba746-console-serving-cert\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.167501 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.167508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffdc38d6-64a0-4164-88a8-459e2c5ba746-trusted-ca-bundle\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.167714 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.167535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffdc38d6-64a0-4164-88a8-459e2c5ba746-console-oauth-config\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.167714 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.167663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffdc38d6-64a0-4164-88a8-459e2c5ba746-console-config\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.167781 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.167720 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffdc38d6-64a0-4164-88a8-459e2c5ba746-oauth-serving-cert\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.167825 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.167777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffdc38d6-64a0-4164-88a8-459e2c5ba746-service-ca\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.167865 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.167831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl85h\" (UniqueName: \"kubernetes.io/projected/ffdc38d6-64a0-4164-88a8-459e2c5ba746-kube-api-access-dl85h\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.168484 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.168455 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffdc38d6-64a0-4164-88a8-459e2c5ba746-console-config\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.168648 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.168516 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffdc38d6-64a0-4164-88a8-459e2c5ba746-trusted-ca-bundle\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.168648 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.168520 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffdc38d6-64a0-4164-88a8-459e2c5ba746-oauth-serving-cert\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.168648 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.168612 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffdc38d6-64a0-4164-88a8-459e2c5ba746-service-ca\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.170468 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.170446 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffdc38d6-64a0-4164-88a8-459e2c5ba746-console-oauth-config\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.170576 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.170561 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffdc38d6-64a0-4164-88a8-459e2c5ba746-console-serving-cert\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.176002 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.175983 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl85h\" (UniqueName: \"kubernetes.io/projected/ffdc38d6-64a0-4164-88a8-459e2c5ba746-kube-api-access-dl85h\") pod \"console-879476b9f-vc6q4\" (UID: \"ffdc38d6-64a0-4164-88a8-459e2c5ba746\") " pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.319100 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.319021 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:38.593196 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:38.593129 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-879476b9f-vc6q4"] Sep 08 10:07:38.596740 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:07:38.596715 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffdc38d6_64a0_4164_88a8_459e2c5ba746.slice/crio-166dcf7cc0da4f894be67e5a0887458f7b68bce192d7b72831bb54a086f1fa9f WatchSource:0}: Error finding container 166dcf7cc0da4f894be67e5a0887458f7b68bce192d7b72831bb54a086f1fa9f: Status 404 returned error can't find the container with id 166dcf7cc0da4f894be67e5a0887458f7b68bce192d7b72831bb54a086f1fa9f Sep 08 10:07:39.263119 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:39.263087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-879476b9f-vc6q4" event={"ID":"ffdc38d6-64a0-4164-88a8-459e2c5ba746","Type":"ContainerStarted","Data":"d54b6458e93571e27370b3ff192a78282162b43f343c87b489b290f1850147c5"} Sep 08 10:07:39.263488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:39.263139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-879476b9f-vc6q4" event={"ID":"ffdc38d6-64a0-4164-88a8-459e2c5ba746","Type":"ContainerStarted","Data":"166dcf7cc0da4f894be67e5a0887458f7b68bce192d7b72831bb54a086f1fa9f"} Sep 08 10:07:39.279603 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:39.279550 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-879476b9f-vc6q4" podStartSLOduration=2.279531169 podStartE2EDuration="2.279531169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:07:39.279005969 +0000 UTC m=+369.068057774" watchObservedRunningTime="2026-09-08 10:07:39.279531169 +0000 UTC m=+369.068582974" Sep 08 10:07:48.319706 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:48.319673 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:48.320123 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:48.319727 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:48.324631 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:48.324590 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:49.312915 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:49.312890 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-879476b9f-vc6q4" Sep 08 10:07:49.461539 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:49.461504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:49.522216 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:49.522191 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:07:49.590409 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:07:49.590344 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:02.232186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:02.232148 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wch2r" Sep 08 10:08:02.235191 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:02.235167 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-25nft" Sep 08 10:08:07.846035 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:07.845975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:07.915460 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:07.915424 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:17.225767 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:17.225731 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:17.319582 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:17.319549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:20.541570 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.541535 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b"] Sep 08 10:08:20.547759 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.545735 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" Sep 08 10:08:20.548879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.548857 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vlcsl\"" Sep 08 10:08:20.549013 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.548865 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 10:08:20.549351 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.549334 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 10:08:20.552245 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.552221 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b"] Sep 08 10:08:20.650941 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.650905 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf"] Sep 08 10:08:20.653208 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.653193 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" Sep 08 10:08:20.662171 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.662139 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf"] Sep 08 10:08:20.729354 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.729325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35cfb63b-6f1a-4ee9-8336-3a21335359a8-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b\" (UID: \"35cfb63b-6f1a-4ee9-8336-3a21335359a8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" Sep 08 10:08:20.729517 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.729375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-966hl\" (UniqueName: \"kubernetes.io/projected/35cfb63b-6f1a-4ee9-8336-3a21335359a8-kube-api-access-966hl\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b\" (UID: \"35cfb63b-6f1a-4ee9-8336-3a21335359a8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" Sep 08 10:08:20.729517 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.729408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35cfb63b-6f1a-4ee9-8336-3a21335359a8-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b\" (UID: \"35cfb63b-6f1a-4ee9-8336-3a21335359a8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" Sep 08 10:08:20.749804 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.749775 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r"] Sep 08 10:08:20.752120 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.752105 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" Sep 08 10:08:20.760641 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.760597 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r"] Sep 08 10:08:20.830430 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.830342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35cfb63b-6f1a-4ee9-8336-3a21335359a8-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b\" (UID: \"35cfb63b-6f1a-4ee9-8336-3a21335359a8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" Sep 08 10:08:20.830430 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.830400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-966hl\" (UniqueName: \"kubernetes.io/projected/35cfb63b-6f1a-4ee9-8336-3a21335359a8-kube-api-access-966hl\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b\" (UID: \"35cfb63b-6f1a-4ee9-8336-3a21335359a8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" Sep 08 10:08:20.830430 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.830426 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8990a85-1456-4a91-9372-f20839634d33-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf\" (UID: \"b8990a85-1456-4a91-9372-f20839634d33\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" Sep 08 10:08:20.830705 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.830445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35cfb63b-6f1a-4ee9-8336-3a21335359a8-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b\" (UID: \"35cfb63b-6f1a-4ee9-8336-3a21335359a8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" Sep 08 10:08:20.830705 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.830475 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8990a85-1456-4a91-9372-f20839634d33-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf\" (UID: \"b8990a85-1456-4a91-9372-f20839634d33\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" Sep 08 10:08:20.830705 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.830491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndfss\" (UniqueName: \"kubernetes.io/projected/b8990a85-1456-4a91-9372-f20839634d33-kube-api-access-ndfss\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf\" (UID: \"b8990a85-1456-4a91-9372-f20839634d33\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" Sep 08 10:08:20.830819 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.830802 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35cfb63b-6f1a-4ee9-8336-3a21335359a8-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b\" (UID: \"35cfb63b-6f1a-4ee9-8336-3a21335359a8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" Sep 08 10:08:20.830868 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.830828 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35cfb63b-6f1a-4ee9-8336-3a21335359a8-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b\" (UID: \"35cfb63b-6f1a-4ee9-8336-3a21335359a8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" Sep 08 10:08:20.838510 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.838481 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-966hl\" (UniqueName: \"kubernetes.io/projected/35cfb63b-6f1a-4ee9-8336-3a21335359a8-kube-api-access-966hl\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b\" (UID: \"35cfb63b-6f1a-4ee9-8336-3a21335359a8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" Sep 08 10:08:20.844927 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.844894 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk"] Sep 08 10:08:20.847726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.847706 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" Sep 08 10:08:20.857043 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.857021 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk"] Sep 08 10:08:20.857763 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.857748 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" Sep 08 10:08:20.931153 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.931120 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a06ed513-247b-4326-86b3-b5a9f2a0bd86-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r\" (UID: \"a06ed513-247b-4326-86b3-b5a9f2a0bd86\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" Sep 08 10:08:20.931153 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.931156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd8mn\" (UniqueName: \"kubernetes.io/projected/a06ed513-247b-4326-86b3-b5a9f2a0bd86-kube-api-access-gd8mn\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r\" (UID: \"a06ed513-247b-4326-86b3-b5a9f2a0bd86\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" Sep 08 10:08:20.931382 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.931190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a06ed513-247b-4326-86b3-b5a9f2a0bd86-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r\" (UID: \"a06ed513-247b-4326-86b3-b5a9f2a0bd86\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" Sep 08 10:08:20.931382 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.931336 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8990a85-1456-4a91-9372-f20839634d33-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf\" (UID: \"b8990a85-1456-4a91-9372-f20839634d33\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" Sep 08 10:08:20.931382 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.931358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8990a85-1456-4a91-9372-f20839634d33-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf\" (UID: \"b8990a85-1456-4a91-9372-f20839634d33\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" Sep 08 10:08:20.931382 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.931376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndfss\" (UniqueName: \"kubernetes.io/projected/b8990a85-1456-4a91-9372-f20839634d33-kube-api-access-ndfss\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf\" (UID: \"b8990a85-1456-4a91-9372-f20839634d33\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" Sep 08 10:08:20.931769 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.931748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8990a85-1456-4a91-9372-f20839634d33-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf\" (UID: \"b8990a85-1456-4a91-9372-f20839634d33\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" Sep 08 10:08:20.931860 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.931783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8990a85-1456-4a91-9372-f20839634d33-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf\" (UID: \"b8990a85-1456-4a91-9372-f20839634d33\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" Sep 08 10:08:20.939350 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.939323 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndfss\" (UniqueName: \"kubernetes.io/projected/b8990a85-1456-4a91-9372-f20839634d33-kube-api-access-ndfss\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf\" (UID: \"b8990a85-1456-4a91-9372-f20839634d33\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" Sep 08 10:08:20.963164 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:20.963137 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" Sep 08 10:08:21.032318 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.032290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk\" (UID: \"4b0b3e6f-64d5-4472-8d82-0918c99ffa15\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" Sep 08 10:08:21.032454 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.032348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a06ed513-247b-4326-86b3-b5a9f2a0bd86-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r\" (UID: \"a06ed513-247b-4326-86b3-b5a9f2a0bd86\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" Sep 08 10:08:21.032454 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.032376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd8mn\" (UniqueName: \"kubernetes.io/projected/a06ed513-247b-4326-86b3-b5a9f2a0bd86-kube-api-access-gd8mn\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r\" (UID: \"a06ed513-247b-4326-86b3-b5a9f2a0bd86\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" Sep 08 10:08:21.032454 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.032411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a06ed513-247b-4326-86b3-b5a9f2a0bd86-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r\" (UID: \"a06ed513-247b-4326-86b3-b5a9f2a0bd86\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" Sep 08 10:08:21.032454 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.032436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk\" (UID: \"4b0b3e6f-64d5-4472-8d82-0918c99ffa15\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" Sep 08 10:08:21.032673 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.032471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rlhp\" (UniqueName: \"kubernetes.io/projected/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-kube-api-access-2rlhp\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk\" (UID: \"4b0b3e6f-64d5-4472-8d82-0918c99ffa15\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" Sep 08 10:08:21.032725 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.032707 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a06ed513-247b-4326-86b3-b5a9f2a0bd86-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r\" (UID: \"a06ed513-247b-4326-86b3-b5a9f2a0bd86\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" Sep 08 10:08:21.032774 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.032744 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a06ed513-247b-4326-86b3-b5a9f2a0bd86-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r\" (UID: \"a06ed513-247b-4326-86b3-b5a9f2a0bd86\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" Sep 08 10:08:21.041273 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.041248 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd8mn\" (UniqueName: \"kubernetes.io/projected/a06ed513-247b-4326-86b3-b5a9f2a0bd86-kube-api-access-gd8mn\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r\" (UID: \"a06ed513-247b-4326-86b3-b5a9f2a0bd86\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" Sep 08 10:08:21.062037 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.062009 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" Sep 08 10:08:21.133849 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.133486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk\" (UID: \"4b0b3e6f-64d5-4472-8d82-0918c99ffa15\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" Sep 08 10:08:21.133849 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.133599 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk\" (UID: \"4b0b3e6f-64d5-4472-8d82-0918c99ffa15\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" Sep 08 10:08:21.133849 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.133665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlhp\" (UniqueName: \"kubernetes.io/projected/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-kube-api-access-2rlhp\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk\" (UID: \"4b0b3e6f-64d5-4472-8d82-0918c99ffa15\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" Sep 08 10:08:21.134136 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.133989 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk\" (UID: \"4b0b3e6f-64d5-4472-8d82-0918c99ffa15\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" Sep 08 10:08:21.134513 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.134273 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk\" (UID: \"4b0b3e6f-64d5-4472-8d82-0918c99ffa15\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" Sep 08 10:08:21.141197 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.141176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rlhp\" (UniqueName: \"kubernetes.io/projected/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-kube-api-access-2rlhp\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk\" (UID: \"4b0b3e6f-64d5-4472-8d82-0918c99ffa15\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" Sep 08 10:08:21.147192 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.147170 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b"] Sep 08 10:08:21.149871 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:08:21.149848 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cfb63b_6f1a_4ee9_8336_3a21335359a8.slice/crio-feb3bf41bcd2315aabfe7bc9f00a4194750bf5bd1c2d49f001b5e819f7b1ba6c WatchSource:0}: Error finding container feb3bf41bcd2315aabfe7bc9f00a4194750bf5bd1c2d49f001b5e819f7b1ba6c: Status 404 returned error can't find the container with id feb3bf41bcd2315aabfe7bc9f00a4194750bf5bd1c2d49f001b5e819f7b1ba6c Sep 08 10:08:21.156677 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.156658 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" Sep 08 10:08:21.239526 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.239396 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf"] Sep 08 10:08:21.244004 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:08:21.243972 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8990a85_1456_4a91_9372_f20839634d33.slice/crio-9f6842fa1aa03f3e693977f1a02e58828218a8e1ce583c329e12b6f662ae42e1 WatchSource:0}: Error finding container 9f6842fa1aa03f3e693977f1a02e58828218a8e1ce583c329e12b6f662ae42e1: Status 404 returned error can't find the container with id 9f6842fa1aa03f3e693977f1a02e58828218a8e1ce583c329e12b6f662ae42e1 Sep 08 10:08:21.333527 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.333506 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r"] Sep 08 10:08:21.345809 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:08:21.345784 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06ed513_247b_4326_86b3_b5a9f2a0bd86.slice/crio-c6b61e7e7755cc46021382b6ce2fc1936be117caab039e7f7f8c08298eb6b1c3 WatchSource:0}: Error finding container c6b61e7e7755cc46021382b6ce2fc1936be117caab039e7f7f8c08298eb6b1c3: Status 404 returned error can't find the container with id c6b61e7e7755cc46021382b6ce2fc1936be117caab039e7f7f8c08298eb6b1c3 Sep 08 10:08:21.437344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.437317 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk"] Sep 08 10:08:21.450879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.450842 2576 generic.go:356] "Generic (PLEG): container finished" podID="a06ed513-247b-4326-86b3-b5a9f2a0bd86" containerID="a8185ceb3dece5aa00dd8f26b3bdd41e8ef0f2b3c31f349766e3ecffe08ab487" exitCode=0 Sep 08 10:08:21.451002 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.450902 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" event={"ID":"a06ed513-247b-4326-86b3-b5a9f2a0bd86","Type":"ContainerDied","Data":"a8185ceb3dece5aa00dd8f26b3bdd41e8ef0f2b3c31f349766e3ecffe08ab487"} Sep 08 10:08:21.451002 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.450955 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" event={"ID":"a06ed513-247b-4326-86b3-b5a9f2a0bd86","Type":"ContainerStarted","Data":"c6b61e7e7755cc46021382b6ce2fc1936be117caab039e7f7f8c08298eb6b1c3"} Sep 08 10:08:21.452726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.452675 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8990a85-1456-4a91-9372-f20839634d33" containerID="dff7aa5b55b62be5492de0c7b71083f3c803549dcf5cdce9746c35f5f59fb2e2" exitCode=0 Sep 08 10:08:21.452726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.452709 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" event={"ID":"b8990a85-1456-4a91-9372-f20839634d33","Type":"ContainerDied","Data":"dff7aa5b55b62be5492de0c7b71083f3c803549dcf5cdce9746c35f5f59fb2e2"} Sep 08 10:08:21.452861 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.452760 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" event={"ID":"b8990a85-1456-4a91-9372-f20839634d33","Type":"ContainerStarted","Data":"9f6842fa1aa03f3e693977f1a02e58828218a8e1ce583c329e12b6f662ae42e1"} Sep 08 10:08:21.454590 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.454567 2576 generic.go:356] "Generic (PLEG): container finished" podID="35cfb63b-6f1a-4ee9-8336-3a21335359a8" containerID="8613686ac507c65f9da9ff8b71ecedb28df6fe0eb587043817e2bf799910124e" exitCode=0 Sep 08 10:08:21.454701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.454645 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" event={"ID":"35cfb63b-6f1a-4ee9-8336-3a21335359a8","Type":"ContainerDied","Data":"8613686ac507c65f9da9ff8b71ecedb28df6fe0eb587043817e2bf799910124e"} Sep 08 10:08:21.454701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:21.454681 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" event={"ID":"35cfb63b-6f1a-4ee9-8336-3a21335359a8","Type":"ContainerStarted","Data":"feb3bf41bcd2315aabfe7bc9f00a4194750bf5bd1c2d49f001b5e819f7b1ba6c"} Sep 08 10:08:21.470409 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:08:21.470388 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0b3e6f_64d5_4472_8d82_0918c99ffa15.slice/crio-64cf44764dc6a2279fc0ddff6bfb2e95f01c77fb457ebb192f2b8f66a62ebdff WatchSource:0}: Error finding container 64cf44764dc6a2279fc0ddff6bfb2e95f01c77fb457ebb192f2b8f66a62ebdff: Status 404 returned error can't find the container with id 64cf44764dc6a2279fc0ddff6bfb2e95f01c77fb457ebb192f2b8f66a62ebdff Sep 08 10:08:22.461547 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:22.461511 2576 generic.go:356] "Generic (PLEG): container finished" podID="a06ed513-247b-4326-86b3-b5a9f2a0bd86" containerID="a7c3678f5f40b7b93eb11d524d90d9b588279223ff524ef74af12a2da4be1e4d" exitCode=0 Sep 08 10:08:22.461950 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:22.461593 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" event={"ID":"a06ed513-247b-4326-86b3-b5a9f2a0bd86","Type":"ContainerDied","Data":"a7c3678f5f40b7b93eb11d524d90d9b588279223ff524ef74af12a2da4be1e4d"} Sep 08 10:08:22.463236 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:22.463206 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b0b3e6f-64d5-4472-8d82-0918c99ffa15" containerID="f6ab7f8e32358274e0651006f3e144789bf9b95e25240fccfa080c4672564565" exitCode=0 Sep 08 10:08:22.463340 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:22.463251 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" event={"ID":"4b0b3e6f-64d5-4472-8d82-0918c99ffa15","Type":"ContainerDied","Data":"f6ab7f8e32358274e0651006f3e144789bf9b95e25240fccfa080c4672564565"} Sep 08 10:08:22.463340 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:22.463294 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" event={"ID":"4b0b3e6f-64d5-4472-8d82-0918c99ffa15","Type":"ContainerStarted","Data":"64cf44764dc6a2279fc0ddff6bfb2e95f01c77fb457ebb192f2b8f66a62ebdff"} Sep 08 10:08:22.465389 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:22.465361 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8990a85-1456-4a91-9372-f20839634d33" containerID="4a44db9df4010f787a0681c07d7f4e3631b6ba2e8283eab54c6fecf330761ad5" exitCode=0 Sep 08 10:08:22.465493 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:22.465450 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" event={"ID":"b8990a85-1456-4a91-9372-f20839634d33","Type":"ContainerDied","Data":"4a44db9df4010f787a0681c07d7f4e3631b6ba2e8283eab54c6fecf330761ad5"} Sep 08 10:08:23.471756 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:23.471716 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8990a85-1456-4a91-9372-f20839634d33" containerID="10251eae2ea149ba67961abe518f00e8e6d0743e220f3f5e65c0faa625eafc46" exitCode=0 Sep 08 10:08:23.472163 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:23.471801 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" event={"ID":"b8990a85-1456-4a91-9372-f20839634d33","Type":"ContainerDied","Data":"10251eae2ea149ba67961abe518f00e8e6d0743e220f3f5e65c0faa625eafc46"} Sep 08 10:08:23.473464 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:23.473437 2576 generic.go:356] "Generic (PLEG): container finished" podID="35cfb63b-6f1a-4ee9-8336-3a21335359a8" containerID="d3fc89846087444679f9b589c5fb38d7df8913b00714ab95be8d07cec7ffde3b" exitCode=0 Sep 08 10:08:23.473604 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:23.473516 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" event={"ID":"35cfb63b-6f1a-4ee9-8336-3a21335359a8","Type":"ContainerDied","Data":"d3fc89846087444679f9b589c5fb38d7df8913b00714ab95be8d07cec7ffde3b"} Sep 08 10:08:23.475878 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:23.475853 2576 generic.go:356] "Generic (PLEG): container finished" podID="a06ed513-247b-4326-86b3-b5a9f2a0bd86" containerID="5ce0058f631c34fd61709e11141eb62656151d772fbdf4adcae460e98103a1a8" exitCode=0 Sep 08 10:08:23.475967 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:23.475893 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" event={"ID":"a06ed513-247b-4326-86b3-b5a9f2a0bd86","Type":"ContainerDied","Data":"5ce0058f631c34fd61709e11141eb62656151d772fbdf4adcae460e98103a1a8"} Sep 08 10:08:24.482311 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.482278 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b0b3e6f-64d5-4472-8d82-0918c99ffa15" containerID="accab93b42f0674e7e872a61d2961d9e3eea93c0cc8d6e52c85c2d12f482eea3" exitCode=0 Sep 08 10:08:24.482778 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.482377 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" event={"ID":"4b0b3e6f-64d5-4472-8d82-0918c99ffa15","Type":"ContainerDied","Data":"accab93b42f0674e7e872a61d2961d9e3eea93c0cc8d6e52c85c2d12f482eea3"} Sep 08 10:08:24.484708 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.484686 2576 generic.go:356] "Generic (PLEG): container finished" podID="35cfb63b-6f1a-4ee9-8336-3a21335359a8" containerID="62244a86cf245fc02b5dd5fc910761653235c3f9cbb094771b587e82d1ddcb3c" exitCode=0 Sep 08 10:08:24.484795 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.484775 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" event={"ID":"35cfb63b-6f1a-4ee9-8336-3a21335359a8","Type":"ContainerDied","Data":"62244a86cf245fc02b5dd5fc910761653235c3f9cbb094771b587e82d1ddcb3c"} Sep 08 10:08:24.627047 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.627023 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" Sep 08 10:08:24.661168 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.661137 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd8mn\" (UniqueName: \"kubernetes.io/projected/a06ed513-247b-4326-86b3-b5a9f2a0bd86-kube-api-access-gd8mn\") pod \"a06ed513-247b-4326-86b3-b5a9f2a0bd86\" (UID: \"a06ed513-247b-4326-86b3-b5a9f2a0bd86\") " Sep 08 10:08:24.661314 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.661206 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a06ed513-247b-4326-86b3-b5a9f2a0bd86-bundle\") pod \"a06ed513-247b-4326-86b3-b5a9f2a0bd86\" (UID: \"a06ed513-247b-4326-86b3-b5a9f2a0bd86\") " Sep 08 10:08:24.661314 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.661225 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a06ed513-247b-4326-86b3-b5a9f2a0bd86-util\") pod \"a06ed513-247b-4326-86b3-b5a9f2a0bd86\" (UID: \"a06ed513-247b-4326-86b3-b5a9f2a0bd86\") " Sep 08 10:08:24.661802 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.661774 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06ed513-247b-4326-86b3-b5a9f2a0bd86-bundle" (OuterVolumeSpecName: "bundle") pod "a06ed513-247b-4326-86b3-b5a9f2a0bd86" (UID: "a06ed513-247b-4326-86b3-b5a9f2a0bd86"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:08:24.663227 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.663209 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06ed513-247b-4326-86b3-b5a9f2a0bd86-kube-api-access-gd8mn" (OuterVolumeSpecName: "kube-api-access-gd8mn") pod "a06ed513-247b-4326-86b3-b5a9f2a0bd86" (UID: "a06ed513-247b-4326-86b3-b5a9f2a0bd86"). InnerVolumeSpecName "kube-api-access-gd8mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:08:24.666756 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.666733 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06ed513-247b-4326-86b3-b5a9f2a0bd86-util" (OuterVolumeSpecName: "util") pod "a06ed513-247b-4326-86b3-b5a9f2a0bd86" (UID: "a06ed513-247b-4326-86b3-b5a9f2a0bd86"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:08:24.695957 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.695931 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" Sep 08 10:08:24.762072 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.761981 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8990a85-1456-4a91-9372-f20839634d33-util\") pod \"b8990a85-1456-4a91-9372-f20839634d33\" (UID: \"b8990a85-1456-4a91-9372-f20839634d33\") " Sep 08 10:08:24.762072 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.762058 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndfss\" (UniqueName: \"kubernetes.io/projected/b8990a85-1456-4a91-9372-f20839634d33-kube-api-access-ndfss\") pod \"b8990a85-1456-4a91-9372-f20839634d33\" (UID: \"b8990a85-1456-4a91-9372-f20839634d33\") " Sep 08 10:08:24.762365 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.762346 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a06ed513-247b-4326-86b3-b5a9f2a0bd86-bundle\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:08:24.762426 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.762370 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a06ed513-247b-4326-86b3-b5a9f2a0bd86-util\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:08:24.762426 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.762386 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd8mn\" (UniqueName: \"kubernetes.io/projected/a06ed513-247b-4326-86b3-b5a9f2a0bd86-kube-api-access-gd8mn\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:08:24.764153 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.764130 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8990a85-1456-4a91-9372-f20839634d33-kube-api-access-ndfss" (OuterVolumeSpecName: "kube-api-access-ndfss") pod "b8990a85-1456-4a91-9372-f20839634d33" (UID: "b8990a85-1456-4a91-9372-f20839634d33"). InnerVolumeSpecName "kube-api-access-ndfss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:08:24.767264 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.767238 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8990a85-1456-4a91-9372-f20839634d33-util" (OuterVolumeSpecName: "util") pod "b8990a85-1456-4a91-9372-f20839634d33" (UID: "b8990a85-1456-4a91-9372-f20839634d33"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:08:24.862725 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.862673 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8990a85-1456-4a91-9372-f20839634d33-bundle\") pod \"b8990a85-1456-4a91-9372-f20839634d33\" (UID: \"b8990a85-1456-4a91-9372-f20839634d33\") " Sep 08 10:08:24.862911 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.862894 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndfss\" (UniqueName: \"kubernetes.io/projected/b8990a85-1456-4a91-9372-f20839634d33-kube-api-access-ndfss\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:08:24.862966 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.862913 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8990a85-1456-4a91-9372-f20839634d33-util\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:08:24.863121 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.863098 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8990a85-1456-4a91-9372-f20839634d33-bundle" (OuterVolumeSpecName: "bundle") pod "b8990a85-1456-4a91-9372-f20839634d33" (UID: "b8990a85-1456-4a91-9372-f20839634d33"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:08:24.963838 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:24.963798 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8990a85-1456-4a91-9372-f20839634d33-bundle\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:08:25.491651 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.491610 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" Sep 08 10:08:25.491651 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.491645 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0m2frf" event={"ID":"b8990a85-1456-4a91-9372-f20839634d33","Type":"ContainerDied","Data":"9f6842fa1aa03f3e693977f1a02e58828218a8e1ce583c329e12b6f662ae42e1"} Sep 08 10:08:25.492117 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.491729 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f6842fa1aa03f3e693977f1a02e58828218a8e1ce583c329e12b6f662ae42e1" Sep 08 10:08:25.493591 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.493572 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" Sep 08 10:08:25.493739 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.493566 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch282r" event={"ID":"a06ed513-247b-4326-86b3-b5a9f2a0bd86","Type":"ContainerDied","Data":"c6b61e7e7755cc46021382b6ce2fc1936be117caab039e7f7f8c08298eb6b1c3"} Sep 08 10:08:25.493739 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.493657 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6b61e7e7755cc46021382b6ce2fc1936be117caab039e7f7f8c08298eb6b1c3" Sep 08 10:08:25.495878 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.495858 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b0b3e6f-64d5-4472-8d82-0918c99ffa15" containerID="29508a7a9aefe7e7fd0209b4822b80f95b1a2705001ec260b6e1c66979f5f014" exitCode=0 Sep 08 10:08:25.495955 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.495886 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" event={"ID":"4b0b3e6f-64d5-4472-8d82-0918c99ffa15","Type":"ContainerDied","Data":"29508a7a9aefe7e7fd0209b4822b80f95b1a2705001ec260b6e1c66979f5f014"} Sep 08 10:08:25.626174 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.626151 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" Sep 08 10:08:25.667865 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.667840 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-966hl\" (UniqueName: \"kubernetes.io/projected/35cfb63b-6f1a-4ee9-8336-3a21335359a8-kube-api-access-966hl\") pod \"35cfb63b-6f1a-4ee9-8336-3a21335359a8\" (UID: \"35cfb63b-6f1a-4ee9-8336-3a21335359a8\") " Sep 08 10:08:25.667985 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.667879 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35cfb63b-6f1a-4ee9-8336-3a21335359a8-util\") pod \"35cfb63b-6f1a-4ee9-8336-3a21335359a8\" (UID: \"35cfb63b-6f1a-4ee9-8336-3a21335359a8\") " Sep 08 10:08:25.667985 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.667927 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35cfb63b-6f1a-4ee9-8336-3a21335359a8-bundle\") pod \"35cfb63b-6f1a-4ee9-8336-3a21335359a8\" (UID: \"35cfb63b-6f1a-4ee9-8336-3a21335359a8\") " Sep 08 10:08:25.668647 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.668599 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cfb63b-6f1a-4ee9-8336-3a21335359a8-bundle" (OuterVolumeSpecName: "bundle") pod "35cfb63b-6f1a-4ee9-8336-3a21335359a8" (UID: "35cfb63b-6f1a-4ee9-8336-3a21335359a8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:08:25.669885 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.669859 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cfb63b-6f1a-4ee9-8336-3a21335359a8-kube-api-access-966hl" (OuterVolumeSpecName: "kube-api-access-966hl") pod "35cfb63b-6f1a-4ee9-8336-3a21335359a8" (UID: "35cfb63b-6f1a-4ee9-8336-3a21335359a8"). InnerVolumeSpecName "kube-api-access-966hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:08:25.675137 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.675112 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cfb63b-6f1a-4ee9-8336-3a21335359a8-util" (OuterVolumeSpecName: "util") pod "35cfb63b-6f1a-4ee9-8336-3a21335359a8" (UID: "35cfb63b-6f1a-4ee9-8336-3a21335359a8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:08:25.768991 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.768915 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35cfb63b-6f1a-4ee9-8336-3a21335359a8-bundle\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:08:25.768991 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.768938 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-966hl\" (UniqueName: \"kubernetes.io/projected/35cfb63b-6f1a-4ee9-8336-3a21335359a8-kube-api-access-966hl\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:08:25.768991 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:25.768953 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35cfb63b-6f1a-4ee9-8336-3a21335359a8-util\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:08:26.502855 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:26.502823 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" Sep 08 10:08:26.503249 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:26.502858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cgh9b" event={"ID":"35cfb63b-6f1a-4ee9-8336-3a21335359a8","Type":"ContainerDied","Data":"feb3bf41bcd2315aabfe7bc9f00a4194750bf5bd1c2d49f001b5e819f7b1ba6c"} Sep 08 10:08:26.503249 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:26.502905 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="feb3bf41bcd2315aabfe7bc9f00a4194750bf5bd1c2d49f001b5e819f7b1ba6c" Sep 08 10:08:26.635208 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:26.635184 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" Sep 08 10:08:26.776878 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:26.776785 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-bundle\") pod \"4b0b3e6f-64d5-4472-8d82-0918c99ffa15\" (UID: \"4b0b3e6f-64d5-4472-8d82-0918c99ffa15\") " Sep 08 10:08:26.776878 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:26.776871 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-util\") pod \"4b0b3e6f-64d5-4472-8d82-0918c99ffa15\" (UID: \"4b0b3e6f-64d5-4472-8d82-0918c99ffa15\") " Sep 08 10:08:26.777053 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:26.776902 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rlhp\" (UniqueName: \"kubernetes.io/projected/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-kube-api-access-2rlhp\") pod \"4b0b3e6f-64d5-4472-8d82-0918c99ffa15\" (UID: \"4b0b3e6f-64d5-4472-8d82-0918c99ffa15\") " Sep 08 10:08:26.777431 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:26.777402 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-bundle" (OuterVolumeSpecName: "bundle") pod "4b0b3e6f-64d5-4472-8d82-0918c99ffa15" (UID: "4b0b3e6f-64d5-4472-8d82-0918c99ffa15"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:08:26.779071 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:26.779046 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-kube-api-access-2rlhp" (OuterVolumeSpecName: "kube-api-access-2rlhp") pod "4b0b3e6f-64d5-4472-8d82-0918c99ffa15" (UID: "4b0b3e6f-64d5-4472-8d82-0918c99ffa15"). InnerVolumeSpecName "kube-api-access-2rlhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:08:26.781403 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:26.781383 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-util" (OuterVolumeSpecName: "util") pod "4b0b3e6f-64d5-4472-8d82-0918c99ffa15" (UID: "4b0b3e6f-64d5-4472-8d82-0918c99ffa15"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:08:26.877803 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:26.877752 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-util\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:08:26.877803 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:26.877797 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rlhp\" (UniqueName: \"kubernetes.io/projected/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-kube-api-access-2rlhp\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:08:26.877803 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:26.877808 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e6f-64d5-4472-8d82-0918c99ffa15-bundle\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:08:27.509935 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:27.509900 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" event={"ID":"4b0b3e6f-64d5-4472-8d82-0918c99ffa15","Type":"ContainerDied","Data":"64cf44764dc6a2279fc0ddff6bfb2e95f01c77fb457ebb192f2b8f66a62ebdff"} Sep 08 10:08:27.510341 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:27.509939 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxtkmk" Sep 08 10:08:27.510341 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:27.509943 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64cf44764dc6a2279fc0ddff6bfb2e95f01c77fb457ebb192f2b8f66a62ebdff" Sep 08 10:08:31.635874 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:31.635817 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:31.733047 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:31.733011 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:33.364888 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.364854 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-bllss"] Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365164 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35cfb63b-6f1a-4ee9-8336-3a21335359a8" containerName="util" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365176 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="35cfb63b-6f1a-4ee9-8336-3a21335359a8" containerName="util" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365186 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a06ed513-247b-4326-86b3-b5a9f2a0bd86" containerName="util" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365191 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a06ed513-247b-4326-86b3-b5a9f2a0bd86" containerName="util" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365196 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8990a85-1456-4a91-9372-f20839634d33" containerName="extract" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365203 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8990a85-1456-4a91-9372-f20839634d33" containerName="extract" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365211 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35cfb63b-6f1a-4ee9-8336-3a21335359a8" containerName="extract" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365216 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="35cfb63b-6f1a-4ee9-8336-3a21335359a8" containerName="extract" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365224 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0b3e6f-64d5-4472-8d82-0918c99ffa15" containerName="pull" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365230 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0b3e6f-64d5-4472-8d82-0918c99ffa15" containerName="pull" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365236 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0b3e6f-64d5-4472-8d82-0918c99ffa15" containerName="extract" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365241 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0b3e6f-64d5-4472-8d82-0918c99ffa15" containerName="extract" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365248 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0b3e6f-64d5-4472-8d82-0918c99ffa15" containerName="util" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365253 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0b3e6f-64d5-4472-8d82-0918c99ffa15" containerName="util" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365260 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8990a85-1456-4a91-9372-f20839634d33" containerName="util" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365265 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8990a85-1456-4a91-9372-f20839634d33" containerName="util" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365272 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35cfb63b-6f1a-4ee9-8336-3a21335359a8" containerName="pull" Sep 08 10:08:33.365270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365277 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="35cfb63b-6f1a-4ee9-8336-3a21335359a8" containerName="pull" Sep 08 10:08:33.365805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365282 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a06ed513-247b-4326-86b3-b5a9f2a0bd86" containerName="pull" Sep 08 10:08:33.365805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365287 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a06ed513-247b-4326-86b3-b5a9f2a0bd86" containerName="pull" Sep 08 10:08:33.365805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365295 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a06ed513-247b-4326-86b3-b5a9f2a0bd86" containerName="extract" Sep 08 10:08:33.365805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365300 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a06ed513-247b-4326-86b3-b5a9f2a0bd86" containerName="extract" Sep 08 10:08:33.365805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365311 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8990a85-1456-4a91-9372-f20839634d33" containerName="pull" Sep 08 10:08:33.365805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365316 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8990a85-1456-4a91-9372-f20839634d33" containerName="pull" Sep 08 10:08:33.365805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365361 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0b3e6f-64d5-4472-8d82-0918c99ffa15" containerName="extract" Sep 08 10:08:33.365805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365368 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8990a85-1456-4a91-9372-f20839634d33" containerName="extract" Sep 08 10:08:33.365805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365374 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a06ed513-247b-4326-86b3-b5a9f2a0bd86" containerName="extract" Sep 08 10:08:33.365805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.365380 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="35cfb63b-6f1a-4ee9-8336-3a21335359a8" containerName="extract" Sep 08 10:08:33.369224 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.369207 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-bllss" Sep 08 10:08:33.371582 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.371559 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 08 10:08:33.371724 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.371703 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-29sjx\"" Sep 08 10:08:33.372150 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.372135 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 08 10:08:33.384693 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.384673 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-bllss"] Sep 08 10:08:33.438483 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.438454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgvlg\" (UniqueName: \"kubernetes.io/projected/4110d091-f0cb-45c6-8350-0ed142047c6c-kube-api-access-bgvlg\") pod \"authorino-operator-68d88948d8-bllss\" (UID: \"4110d091-f0cb-45c6-8350-0ed142047c6c\") " pod="kuadrant-system/authorino-operator-68d88948d8-bllss" Sep 08 10:08:33.539152 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.539124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgvlg\" (UniqueName: \"kubernetes.io/projected/4110d091-f0cb-45c6-8350-0ed142047c6c-kube-api-access-bgvlg\") pod \"authorino-operator-68d88948d8-bllss\" (UID: \"4110d091-f0cb-45c6-8350-0ed142047c6c\") " pod="kuadrant-system/authorino-operator-68d88948d8-bllss" Sep 08 10:08:33.553605 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.553576 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgvlg\" (UniqueName: \"kubernetes.io/projected/4110d091-f0cb-45c6-8350-0ed142047c6c-kube-api-access-bgvlg\") pod \"authorino-operator-68d88948d8-bllss\" (UID: \"4110d091-f0cb-45c6-8350-0ed142047c6c\") " pod="kuadrant-system/authorino-operator-68d88948d8-bllss" Sep 08 10:08:33.679223 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.679193 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-bllss" Sep 08 10:08:33.950947 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:33.950922 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-bllss"] Sep 08 10:08:33.952816 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:08:33.952787 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4110d091_f0cb_45c6_8350_0ed142047c6c.slice/crio-cded0530f916c0d615f25106fdf4d8ec27075debfc031c46f970429a47760ee2 WatchSource:0}: Error finding container cded0530f916c0d615f25106fdf4d8ec27075debfc031c46f970429a47760ee2: Status 404 returned error can't find the container with id cded0530f916c0d615f25106fdf4d8ec27075debfc031c46f970429a47760ee2 Sep 08 10:08:34.521796 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:34.521751 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:34.545133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:34.545099 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-bllss" event={"ID":"4110d091-f0cb-45c6-8350-0ed142047c6c","Type":"ContainerStarted","Data":"cded0530f916c0d615f25106fdf4d8ec27075debfc031c46f970429a47760ee2"} Sep 08 10:08:36.380955 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:36.380919 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj"] Sep 08 10:08:36.390267 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:36.390242 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" Sep 08 10:08:36.393207 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:36.393174 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hmgz9\"" Sep 08 10:08:36.395505 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:36.395446 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj"] Sep 08 10:08:36.465639 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:36.465573 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mmd9\" (UniqueName: \"kubernetes.io/projected/dc0d3e9e-c567-4757-93a0-0760bd9e8a41-kube-api-access-6mmd9\") pod \"limitador-operator-controller-manager-6c549b6d85-mfncj\" (UID: \"dc0d3e9e-c567-4757-93a0-0760bd9e8a41\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" Sep 08 10:08:36.566156 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:36.566119 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mmd9\" (UniqueName: \"kubernetes.io/projected/dc0d3e9e-c567-4757-93a0-0760bd9e8a41-kube-api-access-6mmd9\") pod \"limitador-operator-controller-manager-6c549b6d85-mfncj\" (UID: \"dc0d3e9e-c567-4757-93a0-0760bd9e8a41\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" Sep 08 10:08:36.576424 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:36.576397 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mmd9\" (UniqueName: \"kubernetes.io/projected/dc0d3e9e-c567-4757-93a0-0760bd9e8a41-kube-api-access-6mmd9\") pod \"limitador-operator-controller-manager-6c549b6d85-mfncj\" (UID: \"dc0d3e9e-c567-4757-93a0-0760bd9e8a41\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" Sep 08 10:08:36.703729 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:36.703694 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" Sep 08 10:08:37.207562 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:37.207533 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj"] Sep 08 10:08:37.210637 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:08:37.210589 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0d3e9e_c567_4757_93a0_0760bd9e8a41.slice/crio-e51cd0ce223bf6997e3c03e565e7cbbd945b3553cd01ef3a3d373596ad3fdbdb WatchSource:0}: Error finding container e51cd0ce223bf6997e3c03e565e7cbbd945b3553cd01ef3a3d373596ad3fdbdb: Status 404 returned error can't find the container with id e51cd0ce223bf6997e3c03e565e7cbbd945b3553cd01ef3a3d373596ad3fdbdb Sep 08 10:08:37.564224 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:37.564187 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-bllss" event={"ID":"4110d091-f0cb-45c6-8350-0ed142047c6c","Type":"ContainerStarted","Data":"ee92059cfaabf140301952ca8daa257e4973161087dc6d4cca6a6087903ef965"} Sep 08 10:08:37.564708 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:37.564247 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-bllss" Sep 08 10:08:37.565643 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:37.565612 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" event={"ID":"dc0d3e9e-c567-4757-93a0-0760bd9e8a41","Type":"ContainerStarted","Data":"e51cd0ce223bf6997e3c03e565e7cbbd945b3553cd01ef3a3d373596ad3fdbdb"} Sep 08 10:08:37.738524 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:37.738465 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:39.579119 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:39.579072 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" event={"ID":"dc0d3e9e-c567-4757-93a0-0760bd9e8a41","Type":"ContainerStarted","Data":"dc0eb44e06fd9b6c2807017be98b43c33becf8c12630441c79a9a0ddb1db6263"} Sep 08 10:08:39.579579 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:39.579159 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" Sep 08 10:08:39.599684 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:39.599639 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-bllss" podStartSLOduration=3.559114786 podStartE2EDuration="6.599604176s" totalImagesPullingTime="3.04048939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:08:37.624295717 +0000 UTC m=+427.413347521" watchObservedRunningTime="2026-09-08 10:08:39.599604176 +0000 UTC m=+429.388655974" Sep 08 10:08:39.600422 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:39.600398 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" podStartSLOduration=1.841387415 podStartE2EDuration="3.600391503s" totalImagesPullingTime="1.759004088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:08:39.599087996 +0000 UTC m=+429.388139799" watchObservedRunningTime="2026-09-08 10:08:39.600391503 +0000 UTC m=+429.389443307" Sep 08 10:08:39.656301 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:39.653803 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-mpq95"] Sep 08 10:08:39.658602 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:39.658574 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-mpq95" Sep 08 10:08:39.660898 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:39.660871 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4ctqc\"" Sep 08 10:08:39.661009 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:39.660898 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 08 10:08:39.672372 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:39.672347 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-mpq95"] Sep 08 10:08:39.695287 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:39.695256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ntb\" (UniqueName: \"kubernetes.io/projected/9646f179-ace5-4b54-a40d-fbf644218d46-kube-api-access-s4ntb\") pod \"dns-operator-controller-manager-5985564776-mpq95\" (UID: \"9646f179-ace5-4b54-a40d-fbf644218d46\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-mpq95" Sep 08 10:08:39.796447 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:39.796419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ntb\" (UniqueName: \"kubernetes.io/projected/9646f179-ace5-4b54-a40d-fbf644218d46-kube-api-access-s4ntb\") pod \"dns-operator-controller-manager-5985564776-mpq95\" (UID: \"9646f179-ace5-4b54-a40d-fbf644218d46\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-mpq95" Sep 08 10:08:39.804308 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:39.804277 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ntb\" (UniqueName: \"kubernetes.io/projected/9646f179-ace5-4b54-a40d-fbf644218d46-kube-api-access-s4ntb\") pod \"dns-operator-controller-manager-5985564776-mpq95\" (UID: \"9646f179-ace5-4b54-a40d-fbf644218d46\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-mpq95" Sep 08 10:08:39.968198 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:39.968171 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-mpq95" Sep 08 10:08:40.248599 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:40.248558 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-mpq95"] Sep 08 10:08:40.250877 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:08:40.250847 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9646f179_ace5_4b54_a40d_fbf644218d46.slice/crio-c90165dc0df1eee1485d487a2b8abd5552f2d839f6ef9a4257ad65746abc964a WatchSource:0}: Error finding container c90165dc0df1eee1485d487a2b8abd5552f2d839f6ef9a4257ad65746abc964a: Status 404 returned error can't find the container with id c90165dc0df1eee1485d487a2b8abd5552f2d839f6ef9a4257ad65746abc964a Sep 08 10:08:40.275942 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:40.275918 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:40.588674 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:40.588567 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-mpq95" event={"ID":"9646f179-ace5-4b54-a40d-fbf644218d46","Type":"ContainerStarted","Data":"c90165dc0df1eee1485d487a2b8abd5552f2d839f6ef9a4257ad65746abc964a"} Sep 08 10:08:41.163167 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:41.163133 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:41.450737 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:41.450642 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:42.602504 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:42.602427 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-mpq95" event={"ID":"9646f179-ace5-4b54-a40d-fbf644218d46","Type":"ContainerStarted","Data":"c3721379ac47f55a29ae73cac2e3e382c9aa5b2c9e03a5eb439ae02b2ad3f057"} Sep 08 10:08:42.602890 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:42.602540 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-mpq95" Sep 08 10:08:42.620168 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:42.620113 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-mpq95" podStartSLOduration=1.601090917 podStartE2EDuration="3.620098449s" totalImagesPullingTime="2.019007532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:08:42.619332233 +0000 UTC m=+432.408384038" watchObservedRunningTime="2026-09-08 10:08:42.620098449 +0000 UTC m=+432.409150252" Sep 08 10:08:45.489792 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:45.489755 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf"] Sep 08 10:08:45.493196 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:45.493179 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:08:45.495514 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:45.495492 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pz647\"" Sep 08 10:08:45.508864 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:45.508833 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf"] Sep 08 10:08:45.544840 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:45.544812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpqg7\" (UniqueName: \"kubernetes.io/projected/388f6dd8-ee7b-4173-8697-805c6b001dc1-kube-api-access-mpqg7\") pod \"kuadrant-operator-controller-manager-58fc56757c-xjfdf\" (UID: \"388f6dd8-ee7b-4173-8697-805c6b001dc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:08:45.544966 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:45.544861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/388f6dd8-ee7b-4173-8697-805c6b001dc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-xjfdf\" (UID: \"388f6dd8-ee7b-4173-8697-805c6b001dc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:08:45.645970 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:45.645932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqg7\" (UniqueName: \"kubernetes.io/projected/388f6dd8-ee7b-4173-8697-805c6b001dc1-kube-api-access-mpqg7\") pod \"kuadrant-operator-controller-manager-58fc56757c-xjfdf\" (UID: \"388f6dd8-ee7b-4173-8697-805c6b001dc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:08:45.646125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:45.645990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/388f6dd8-ee7b-4173-8697-805c6b001dc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-xjfdf\" (UID: \"388f6dd8-ee7b-4173-8697-805c6b001dc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:08:45.646375 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:45.646357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/388f6dd8-ee7b-4173-8697-805c6b001dc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-xjfdf\" (UID: \"388f6dd8-ee7b-4173-8697-805c6b001dc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:08:45.654902 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:45.654880 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpqg7\" (UniqueName: \"kubernetes.io/projected/388f6dd8-ee7b-4173-8697-805c6b001dc1-kube-api-access-mpqg7\") pod \"kuadrant-operator-controller-manager-58fc56757c-xjfdf\" (UID: \"388f6dd8-ee7b-4173-8697-805c6b001dc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:08:45.804343 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:45.804257 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:08:46.089417 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:46.089386 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf"] Sep 08 10:08:46.090266 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:08:46.090237 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388f6dd8_ee7b_4173_8697_805c6b001dc1.slice/crio-c321c7fc166d4e238b74af9525c8edeac1795a469729287972a700dd02154c61 WatchSource:0}: Error finding container c321c7fc166d4e238b74af9525c8edeac1795a469729287972a700dd02154c61: Status 404 returned error can't find the container with id c321c7fc166d4e238b74af9525c8edeac1795a469729287972a700dd02154c61 Sep 08 10:08:46.623813 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:46.623780 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" event={"ID":"388f6dd8-ee7b-4173-8697-805c6b001dc1","Type":"ContainerStarted","Data":"c321c7fc166d4e238b74af9525c8edeac1795a469729287972a700dd02154c61"} Sep 08 10:08:48.575073 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:48.575035 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-bllss" Sep 08 10:08:50.592174 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:50.592084 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" Sep 08 10:08:50.658349 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:50.658315 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" event={"ID":"388f6dd8-ee7b-4173-8697-805c6b001dc1","Type":"ContainerStarted","Data":"700f0cf6c903e865a6b5cdef2c0b52ccb29c2134faeb6741b67b42eba496737c"} Sep 08 10:08:50.658543 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:50.658422 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:08:50.682677 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:50.682630 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" podStartSLOduration=1.624600301 podStartE2EDuration="5.682601223s" totalImagesPullingTime="4.058000922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:08:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:08:50.679969821 +0000 UTC m=+440.469021646" watchObservedRunningTime="2026-09-08 10:08:50.682601223 +0000 UTC m=+440.471653027" Sep 08 10:08:52.748993 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:52.748954 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx"] Sep 08 10:08:52.753593 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:52.753569 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx" Sep 08 10:08:52.755701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:52.755674 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 08 10:08:52.755867 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:52.755710 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 08 10:08:52.755867 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:52.755760 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dqpmw\"" Sep 08 10:08:52.760118 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:52.760096 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx"] Sep 08 10:08:52.915603 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:52.915518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b43829ee-9034-42b6-92fe-9b4f7f66392b-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-b2czx\" (UID: \"b43829ee-9034-42b6-92fe-9b4f7f66392b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx" Sep 08 10:08:52.915603 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:52.915556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hg4s\" (UniqueName: \"kubernetes.io/projected/b43829ee-9034-42b6-92fe-9b4f7f66392b-kube-api-access-7hg4s\") pod \"kuadrant-console-plugin-598f86468b-b2czx\" (UID: \"b43829ee-9034-42b6-92fe-9b4f7f66392b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx" Sep 08 10:08:52.915820 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:52.915683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b43829ee-9034-42b6-92fe-9b4f7f66392b-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-b2czx\" (UID: \"b43829ee-9034-42b6-92fe-9b4f7f66392b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx" Sep 08 10:08:53.016447 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:53.016350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b43829ee-9034-42b6-92fe-9b4f7f66392b-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-b2czx\" (UID: \"b43829ee-9034-42b6-92fe-9b4f7f66392b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx" Sep 08 10:08:53.016447 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:53.016418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b43829ee-9034-42b6-92fe-9b4f7f66392b-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-b2czx\" (UID: \"b43829ee-9034-42b6-92fe-9b4f7f66392b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx" Sep 08 10:08:53.016447 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:53.016439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hg4s\" (UniqueName: \"kubernetes.io/projected/b43829ee-9034-42b6-92fe-9b4f7f66392b-kube-api-access-7hg4s\") pod \"kuadrant-console-plugin-598f86468b-b2czx\" (UID: \"b43829ee-9034-42b6-92fe-9b4f7f66392b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx" Sep 08 10:08:53.017161 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:53.017131 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b43829ee-9034-42b6-92fe-9b4f7f66392b-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-b2czx\" (UID: \"b43829ee-9034-42b6-92fe-9b4f7f66392b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx" Sep 08 10:08:53.018907 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:53.018886 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b43829ee-9034-42b6-92fe-9b4f7f66392b-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-b2czx\" (UID: \"b43829ee-9034-42b6-92fe-9b4f7f66392b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx" Sep 08 10:08:53.024123 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:53.024098 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hg4s\" (UniqueName: \"kubernetes.io/projected/b43829ee-9034-42b6-92fe-9b4f7f66392b-kube-api-access-7hg4s\") pod \"kuadrant-console-plugin-598f86468b-b2czx\" (UID: \"b43829ee-9034-42b6-92fe-9b4f7f66392b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx" Sep 08 10:08:53.064228 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:53.064200 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx" Sep 08 10:08:53.336278 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:53.336207 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx"] Sep 08 10:08:53.338384 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:08:53.338359 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb43829ee_9034_42b6_92fe_9b4f7f66392b.slice/crio-9f865b1d3cda272774eb1d04394c86df82ba1b6769456f1e5995e166cac20e33 WatchSource:0}: Error finding container 9f865b1d3cda272774eb1d04394c86df82ba1b6769456f1e5995e166cac20e33: Status 404 returned error can't find the container with id 9f865b1d3cda272774eb1d04394c86df82ba1b6769456f1e5995e166cac20e33 Sep 08 10:08:53.609780 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:53.609695 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-mpq95" Sep 08 10:08:53.675749 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:53.675704 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx" event={"ID":"b43829ee-9034-42b6-92fe-9b4f7f66392b","Type":"ContainerStarted","Data":"9f865b1d3cda272774eb1d04394c86df82ba1b6769456f1e5995e166cac20e33"} Sep 08 10:08:54.176399 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:54.176361 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:54.329233 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:54.329197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:57.437944 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:57.437909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:57.554120 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:57.554086 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:57.996339 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:57.996302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:58.705433 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:58.705396 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx" event={"ID":"b43829ee-9034-42b6-92fe-9b4f7f66392b","Type":"ContainerStarted","Data":"43be037385417661656ab31492b917b03f581dac48cfd314b7643245e7354484"} Sep 08 10:08:58.773095 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:58.773038 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b2czx" podStartSLOduration=2.300968803 podStartE2EDuration="6.773023269s" totalImagesPullingTime="4.472054466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:08:58.771507174 +0000 UTC m=+448.560558978" watchObservedRunningTime="2026-09-08 10:08:58.773023269 +0000 UTC m=+448.562075099" Sep 08 10:08:58.886400 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:58.886343 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:08:59.008383 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:08:59.008287 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:01.666206 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:01.666171 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:09:01.782690 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:01.782579 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:01.976778 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:01.976698 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:02.733538 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:02.733508 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:02.837557 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:02.837527 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:03.220233 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:03.220188 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m"] Sep 08 10:09:03.224935 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:03.224912 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m" Sep 08 10:09:03.244654 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:03.244612 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m"] Sep 08 10:09:03.307587 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:03.307551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8181c44-35b6-4760-9ef5-622dd4b8bc6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-v7p4m\" (UID: \"e8181c44-35b6-4760-9ef5-622dd4b8bc6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m" Sep 08 10:09:03.307965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:03.307935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76j2x\" (UniqueName: \"kubernetes.io/projected/e8181c44-35b6-4760-9ef5-622dd4b8bc6d-kube-api-access-76j2x\") pod \"kuadrant-operator-controller-manager-6bd956ccf-v7p4m\" (UID: \"e8181c44-35b6-4760-9ef5-622dd4b8bc6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m" Sep 08 10:09:03.378381 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:03.378352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:03.408667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:03.408609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8181c44-35b6-4760-9ef5-622dd4b8bc6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-v7p4m\" (UID: \"e8181c44-35b6-4760-9ef5-622dd4b8bc6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m" Sep 08 10:09:03.408801 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:03.408715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76j2x\" (UniqueName: \"kubernetes.io/projected/e8181c44-35b6-4760-9ef5-622dd4b8bc6d-kube-api-access-76j2x\") pod \"kuadrant-operator-controller-manager-6bd956ccf-v7p4m\" (UID: \"e8181c44-35b6-4760-9ef5-622dd4b8bc6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m" Sep 08 10:09:03.409092 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:03.409055 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8181c44-35b6-4760-9ef5-622dd4b8bc6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-v7p4m\" (UID: \"e8181c44-35b6-4760-9ef5-622dd4b8bc6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m" Sep 08 10:09:03.426002 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:03.425969 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76j2x\" (UniqueName: \"kubernetes.io/projected/e8181c44-35b6-4760-9ef5-622dd4b8bc6d-kube-api-access-76j2x\") pod \"kuadrant-operator-controller-manager-6bd956ccf-v7p4m\" (UID: \"e8181c44-35b6-4760-9ef5-622dd4b8bc6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m" Sep 08 10:09:03.519543 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:03.519454 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:03.534600 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:03.534556 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m" Sep 08 10:09:03.820270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:03.820246 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m"] Sep 08 10:09:03.822293 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:09:03.822260 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8181c44_35b6_4760_9ef5_622dd4b8bc6d.slice/crio-66285baec3f11376ac3b9e29468052e51f2d18b1c6ea969a4452508985c7492e WatchSource:0}: Error finding container 66285baec3f11376ac3b9e29468052e51f2d18b1c6ea969a4452508985c7492e: Status 404 returned error can't find the container with id 66285baec3f11376ac3b9e29468052e51f2d18b1c6ea969a4452508985c7492e Sep 08 10:09:04.176710 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.176676 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf"] Sep 08 10:09:04.176900 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.176878 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" podUID="388f6dd8-ee7b-4173-8697-805c6b001dc1" containerName="manager" containerID="cri-o://700f0cf6c903e865a6b5cdef2c0b52ccb29c2134faeb6741b67b42eba496737c" gracePeriod=2 Sep 08 10:09:04.196006 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.195979 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf"] Sep 08 10:09:04.207852 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:04.207809 2576 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"388f6dd8-ee7b-4173-8697-805c6b001dc1\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-58fc56757c-xjfdf\": pods \"kuadrant-operator-controller-manager-58fc56757c-xjfdf\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:09:04.211559 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.211534 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m"] Sep 08 10:09:04.216035 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.216008 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm"] Sep 08 10:09:04.216502 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.216483 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="388f6dd8-ee7b-4173-8697-805c6b001dc1" containerName="manager" Sep 08 10:09:04.216557 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.216506 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="388f6dd8-ee7b-4173-8697-805c6b001dc1" containerName="manager" Sep 08 10:09:04.216598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.216584 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="388f6dd8-ee7b-4173-8697-805c6b001dc1" containerName="manager" Sep 08 10:09:04.219879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.219860 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" Sep 08 10:09:04.221342 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.221315 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m"] Sep 08 10:09:04.233697 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.233668 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm"] Sep 08 10:09:04.240877 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.240843 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh"] Sep 08 10:09:04.241567 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.241539 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8181c44-35b6-4760-9ef5-622dd4b8bc6d" containerName="manager" Sep 08 10:09:04.241567 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.241569 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8181c44-35b6-4760-9ef5-622dd4b8bc6d" containerName="manager" Sep 08 10:09:04.241713 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.241697 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8181c44-35b6-4760-9ef5-622dd4b8bc6d" containerName="manager" Sep 08 10:09:04.244903 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.244883 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" Sep 08 10:09:04.270799 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:04.270753 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-xjfdf\" is forbidden: User \"system:node:ip-10-0-132-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-206.ec2.internal' and this object" podUID="388f6dd8-ee7b-4173-8697-805c6b001dc1" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:09:04.276128 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.276103 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh"] Sep 08 10:09:04.278597 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.278569 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj"] Sep 08 10:09:04.278858 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.278827 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" podUID="dc0d3e9e-c567-4757-93a0-0760bd9e8a41" containerName="manager" containerID="cri-o://dc0eb44e06fd9b6c2807017be98b43c33becf8c12630441c79a9a0ddb1db6263" gracePeriod=2 Sep 08 10:09:04.287890 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.287857 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj"] Sep 08 10:09:04.300751 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:04.300570 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-xjfdf\" is forbidden: User \"system:node:ip-10-0-132-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-206.ec2.internal' and this object" podUID="388f6dd8-ee7b-4173-8697-805c6b001dc1" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:09:04.302435 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:04.302406 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-mfncj\" is forbidden: User \"system:node:ip-10-0-132-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-206.ec2.internal' and this object" podUID="dc0d3e9e-c567-4757-93a0-0760bd9e8a41" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" Sep 08 10:09:04.304368 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:04.304334 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-xjfdf\" is forbidden: User \"system:node:ip-10-0-132-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-206.ec2.internal' and this object" podUID="388f6dd8-ee7b-4173-8697-805c6b001dc1" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:09:04.317493 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.317465 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/537ac7d4-f6b5-4550-899a-f49f9a411885-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-vd2rm\" (UID: \"537ac7d4-f6b5-4550-899a-f49f9a411885\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" Sep 08 10:09:04.317632 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.317533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vrw8\" (UniqueName: \"kubernetes.io/projected/537ac7d4-f6b5-4550-899a-f49f9a411885-kube-api-access-5vrw8\") pod \"kuadrant-operator-controller-manager-58fc56757c-vd2rm\" (UID: \"537ac7d4-f6b5-4550-899a-f49f9a411885\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" Sep 08 10:09:04.418392 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.418365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/537ac7d4-f6b5-4550-899a-f49f9a411885-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-vd2rm\" (UID: \"537ac7d4-f6b5-4550-899a-f49f9a411885\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" Sep 08 10:09:04.418531 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.418425 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vrw8\" (UniqueName: \"kubernetes.io/projected/537ac7d4-f6b5-4550-899a-f49f9a411885-kube-api-access-5vrw8\") pod \"kuadrant-operator-controller-manager-58fc56757c-vd2rm\" (UID: \"537ac7d4-f6b5-4550-899a-f49f9a411885\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" Sep 08 10:09:04.418531 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.418462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47ad138f-2aa6-4d4e-b471-1fd88c995b64-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-rfgvh\" (UID: \"47ad138f-2aa6-4d4e-b471-1fd88c995b64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" Sep 08 10:09:04.418531 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.418493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfxgl\" (UniqueName: \"kubernetes.io/projected/47ad138f-2aa6-4d4e-b471-1fd88c995b64-kube-api-access-tfxgl\") pod \"kuadrant-operator-controller-manager-6bd956ccf-rfgvh\" (UID: \"47ad138f-2aa6-4d4e-b471-1fd88c995b64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" Sep 08 10:09:04.418809 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.418755 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/537ac7d4-f6b5-4550-899a-f49f9a411885-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-vd2rm\" (UID: \"537ac7d4-f6b5-4550-899a-f49f9a411885\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" Sep 08 10:09:04.419452 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.419429 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:09:04.423097 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:04.423070 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-mfncj\" is forbidden: User \"system:node:ip-10-0-132-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-206.ec2.internal' and this object" podUID="dc0d3e9e-c567-4757-93a0-0760bd9e8a41" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" Sep 08 10:09:04.424997 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:04.424969 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-xjfdf\" is forbidden: User \"system:node:ip-10-0-132-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-206.ec2.internal' and this object" podUID="388f6dd8-ee7b-4173-8697-805c6b001dc1" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:09:04.428183 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.428128 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vrw8\" (UniqueName: \"kubernetes.io/projected/537ac7d4-f6b5-4550-899a-f49f9a411885-kube-api-access-5vrw8\") pod \"kuadrant-operator-controller-manager-58fc56757c-vd2rm\" (UID: \"537ac7d4-f6b5-4550-899a-f49f9a411885\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" Sep 08 10:09:04.519649 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.519608 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpqg7\" (UniqueName: \"kubernetes.io/projected/388f6dd8-ee7b-4173-8697-805c6b001dc1-kube-api-access-mpqg7\") pod \"388f6dd8-ee7b-4173-8697-805c6b001dc1\" (UID: \"388f6dd8-ee7b-4173-8697-805c6b001dc1\") " Sep 08 10:09:04.519786 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.519663 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/388f6dd8-ee7b-4173-8697-805c6b001dc1-extensions-socket-volume\") pod \"388f6dd8-ee7b-4173-8697-805c6b001dc1\" (UID: \"388f6dd8-ee7b-4173-8697-805c6b001dc1\") " Sep 08 10:09:04.519917 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.519900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47ad138f-2aa6-4d4e-b471-1fd88c995b64-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-rfgvh\" (UID: \"47ad138f-2aa6-4d4e-b471-1fd88c995b64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" Sep 08 10:09:04.519992 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.519931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfxgl\" (UniqueName: \"kubernetes.io/projected/47ad138f-2aa6-4d4e-b471-1fd88c995b64-kube-api-access-tfxgl\") pod \"kuadrant-operator-controller-manager-6bd956ccf-rfgvh\" (UID: \"47ad138f-2aa6-4d4e-b471-1fd88c995b64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" Sep 08 10:09:04.520156 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.520126 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388f6dd8-ee7b-4173-8697-805c6b001dc1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "388f6dd8-ee7b-4173-8697-805c6b001dc1" (UID: "388f6dd8-ee7b-4173-8697-805c6b001dc1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:09:04.520304 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.520285 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47ad138f-2aa6-4d4e-b471-1fd88c995b64-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-rfgvh\" (UID: \"47ad138f-2aa6-4d4e-b471-1fd88c995b64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" Sep 08 10:09:04.521612 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.521587 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388f6dd8-ee7b-4173-8697-805c6b001dc1-kube-api-access-mpqg7" (OuterVolumeSpecName: "kube-api-access-mpqg7") pod "388f6dd8-ee7b-4173-8697-805c6b001dc1" (UID: "388f6dd8-ee7b-4173-8697-805c6b001dc1"). InnerVolumeSpecName "kube-api-access-mpqg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:09:04.528777 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.528755 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfxgl\" (UniqueName: \"kubernetes.io/projected/47ad138f-2aa6-4d4e-b471-1fd88c995b64-kube-api-access-tfxgl\") pod \"kuadrant-operator-controller-manager-6bd956ccf-rfgvh\" (UID: \"47ad138f-2aa6-4d4e-b471-1fd88c995b64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" Sep 08 10:09:04.545831 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.545803 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" Sep 08 10:09:04.548023 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:04.547999 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-mfncj\" is forbidden: User \"system:node:ip-10-0-132-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-206.ec2.internal' and this object" podUID="dc0d3e9e-c567-4757-93a0-0760bd9e8a41" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" Sep 08 10:09:04.549988 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:04.549964 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-xjfdf\" is forbidden: User \"system:node:ip-10-0-132-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-206.ec2.internal' and this object" podUID="388f6dd8-ee7b-4173-8697-805c6b001dc1" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:09:04.571873 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.571848 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" Sep 08 10:09:04.579730 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.579705 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" Sep 08 10:09:04.620865 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.620815 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpqg7\" (UniqueName: \"kubernetes.io/projected/388f6dd8-ee7b-4173-8697-805c6b001dc1-kube-api-access-mpqg7\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:09:04.620865 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.620866 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/388f6dd8-ee7b-4173-8697-805c6b001dc1-extensions-socket-volume\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:09:04.678500 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.678424 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="388f6dd8-ee7b-4173-8697-805c6b001dc1" path="/var/lib/kubelet/pods/388f6dd8-ee7b-4173-8697-805c6b001dc1/volumes" Sep 08 10:09:04.721766 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.721725 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mmd9\" (UniqueName: \"kubernetes.io/projected/dc0d3e9e-c567-4757-93a0-0760bd9e8a41-kube-api-access-6mmd9\") pod \"dc0d3e9e-c567-4757-93a0-0760bd9e8a41\" (UID: \"dc0d3e9e-c567-4757-93a0-0760bd9e8a41\") " Sep 08 10:09:04.723813 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.723783 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0d3e9e-c567-4757-93a0-0760bd9e8a41-kube-api-access-6mmd9" (OuterVolumeSpecName: "kube-api-access-6mmd9") pod "dc0d3e9e-c567-4757-93a0-0760bd9e8a41" (UID: "dc0d3e9e-c567-4757-93a0-0760bd9e8a41"). InnerVolumeSpecName "kube-api-access-6mmd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:09:04.739070 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.739037 2576 generic.go:356] "Generic (PLEG): container finished" podID="388f6dd8-ee7b-4173-8697-805c6b001dc1" containerID="700f0cf6c903e865a6b5cdef2c0b52ccb29c2134faeb6741b67b42eba496737c" exitCode=0 Sep 08 10:09:04.739232 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.739083 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xjfdf" Sep 08 10:09:04.739232 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.739096 2576 scope.go:117] "RemoveContainer" containerID="700f0cf6c903e865a6b5cdef2c0b52ccb29c2134faeb6741b67b42eba496737c" Sep 08 10:09:04.740693 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.740666 2576 generic.go:356] "Generic (PLEG): container finished" podID="dc0d3e9e-c567-4757-93a0-0760bd9e8a41" containerID="dc0eb44e06fd9b6c2807017be98b43c33becf8c12630441c79a9a0ddb1db6263" exitCode=0 Sep 08 10:09:04.740791 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.740715 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mfncj" Sep 08 10:09:04.742908 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:04.742892 2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m" container="manager" Sep 08 10:09:04.749089 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.749070 2576 scope.go:117] "RemoveContainer" containerID="700f0cf6c903e865a6b5cdef2c0b52ccb29c2134faeb6741b67b42eba496737c" Sep 08 10:09:04.749408 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:04.749384 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"700f0cf6c903e865a6b5cdef2c0b52ccb29c2134faeb6741b67b42eba496737c\": container with ID starting with 700f0cf6c903e865a6b5cdef2c0b52ccb29c2134faeb6741b67b42eba496737c not found: ID does not exist" containerID="700f0cf6c903e865a6b5cdef2c0b52ccb29c2134faeb6741b67b42eba496737c" Sep 08 10:09:04.749499 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.749421 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"700f0cf6c903e865a6b5cdef2c0b52ccb29c2134faeb6741b67b42eba496737c"} err="failed to get container status \"700f0cf6c903e865a6b5cdef2c0b52ccb29c2134faeb6741b67b42eba496737c\": rpc error: code = NotFound desc = could not find container \"700f0cf6c903e865a6b5cdef2c0b52ccb29c2134faeb6741b67b42eba496737c\": container with ID starting with 700f0cf6c903e865a6b5cdef2c0b52ccb29c2134faeb6741b67b42eba496737c not found: ID does not exist" Sep 08 10:09:04.749499 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.749445 2576 scope.go:117] "RemoveContainer" containerID="dc0eb44e06fd9b6c2807017be98b43c33becf8c12630441c79a9a0ddb1db6263" Sep 08 10:09:04.758635 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.758601 2576 scope.go:117] "RemoveContainer" containerID="dc0eb44e06fd9b6c2807017be98b43c33becf8c12630441c79a9a0ddb1db6263" Sep 08 10:09:04.758942 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:04.758925 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0eb44e06fd9b6c2807017be98b43c33becf8c12630441c79a9a0ddb1db6263\": container with ID starting with dc0eb44e06fd9b6c2807017be98b43c33becf8c12630441c79a9a0ddb1db6263 not found: ID does not exist" containerID="dc0eb44e06fd9b6c2807017be98b43c33becf8c12630441c79a9a0ddb1db6263" Sep 08 10:09:04.759003 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.758952 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0eb44e06fd9b6c2807017be98b43c33becf8c12630441c79a9a0ddb1db6263"} err="failed to get container status \"dc0eb44e06fd9b6c2807017be98b43c33becf8c12630441c79a9a0ddb1db6263\": rpc error: code = NotFound desc = could not find container \"dc0eb44e06fd9b6c2807017be98b43c33becf8c12630441c79a9a0ddb1db6263\": container with ID starting with dc0eb44e06fd9b6c2807017be98b43c33becf8c12630441c79a9a0ddb1db6263 not found: ID does not exist" Sep 08 10:09:04.822398 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.822361 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mmd9\" (UniqueName: \"kubernetes.io/projected/dc0d3e9e-c567-4757-93a0-0760bd9e8a41-kube-api-access-6mmd9\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:09:04.860503 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.860474 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm"] Sep 08 10:09:04.862515 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:09:04.862486 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537ac7d4_f6b5_4550_899a_f49f9a411885.slice/crio-974aa42d2b5dd232ad558d8b654900dabaafde473b9eb7d01f41e3eb49d2e296 WatchSource:0}: Error finding container 974aa42d2b5dd232ad558d8b654900dabaafde473b9eb7d01f41e3eb49d2e296: Status 404 returned error can't find the container with id 974aa42d2b5dd232ad558d8b654900dabaafde473b9eb7d01f41e3eb49d2e296 Sep 08 10:09:04.863964 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:04.863944 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh"] Sep 08 10:09:04.865803 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:09:04.865778 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ad138f_2aa6_4d4e_b471_1fd88c995b64.slice/crio-62c821869287fe4c378afb2fda856f2fca3fd4566cb7dce86644967beda24494 WatchSource:0}: Error finding container 62c821869287fe4c378afb2fda856f2fca3fd4566cb7dce86644967beda24494: Status 404 returned error can't find the container with id 62c821869287fe4c378afb2fda856f2fca3fd4566cb7dce86644967beda24494 Sep 08 10:09:05.749384 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:05.749294 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" event={"ID":"537ac7d4-f6b5-4550-899a-f49f9a411885","Type":"ContainerStarted","Data":"215f0729d540b1dba2c274324e37f541b42b2c9ffc708f19e64fbaa69b5a8f6d"} Sep 08 10:09:05.749384 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:05.749349 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" event={"ID":"537ac7d4-f6b5-4550-899a-f49f9a411885","Type":"ContainerStarted","Data":"974aa42d2b5dd232ad558d8b654900dabaafde473b9eb7d01f41e3eb49d2e296"} Sep 08 10:09:05.749588 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:05.749457 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" Sep 08 10:09:05.751244 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:05.751205 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m" podUID="e8181c44-35b6-4760-9ef5-622dd4b8bc6d" containerName="manager" containerID="cri-o://85042b39f2b9a9848d8d83b014eea0b000618ab9b951faf9cfea8813803c3a00" gracePeriod=2 Sep 08 10:09:05.752904 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:05.752873 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" event={"ID":"47ad138f-2aa6-4d4e-b471-1fd88c995b64","Type":"ContainerStarted","Data":"6961eeeb23354dda5deeda9d375aea183fa5e423df6d1de271a47b453e220470"} Sep 08 10:09:05.753023 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:05.752929 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" Sep 08 10:09:05.753023 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:05.752954 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" event={"ID":"47ad138f-2aa6-4d4e-b471-1fd88c995b64","Type":"ContainerStarted","Data":"62c821869287fe4c378afb2fda856f2fca3fd4566cb7dce86644967beda24494"} Sep 08 10:09:05.770129 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:05.770087 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" podStartSLOduration=1.7700701840000002 podStartE2EDuration="1.770070184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:09:05.768644082 +0000 UTC m=+455.557695878" watchObservedRunningTime="2026-09-08 10:09:05.770070184 +0000 UTC m=+455.559121987" Sep 08 10:09:05.771403 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:05.771376 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-v7p4m\" is forbidden: User \"system:node:ip-10-0-132-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-206.ec2.internal' and this object" podUID="e8181c44-35b6-4760-9ef5-622dd4b8bc6d" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m" Sep 08 10:09:05.793908 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:05.793845 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" podStartSLOduration=1.793825176 podStartE2EDuration="1.793825176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:09:05.792286877 +0000 UTC m=+455.581338707" watchObservedRunningTime="2026-09-08 10:09:05.793825176 +0000 UTC m=+455.582876979" Sep 08 10:09:05.998178 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:05.998139 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m" Sep 08 10:09:06.000961 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:06.000882 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-v7p4m\" is forbidden: User \"system:node:ip-10-0-132-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-206.ec2.internal' and this object" podUID="e8181c44-35b6-4760-9ef5-622dd4b8bc6d" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m" Sep 08 10:09:06.135147 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:06.135091 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8181c44-35b6-4760-9ef5-622dd4b8bc6d-extensions-socket-volume\") pod \"e8181c44-35b6-4760-9ef5-622dd4b8bc6d\" (UID: \"e8181c44-35b6-4760-9ef5-622dd4b8bc6d\") " Sep 08 10:09:06.135360 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:06.135171 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76j2x\" (UniqueName: \"kubernetes.io/projected/e8181c44-35b6-4760-9ef5-622dd4b8bc6d-kube-api-access-76j2x\") pod \"e8181c44-35b6-4760-9ef5-622dd4b8bc6d\" (UID: \"e8181c44-35b6-4760-9ef5-622dd4b8bc6d\") " Sep 08 10:09:06.135567 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:06.135530 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8181c44-35b6-4760-9ef5-622dd4b8bc6d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e8181c44-35b6-4760-9ef5-622dd4b8bc6d" (UID: "e8181c44-35b6-4760-9ef5-622dd4b8bc6d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:09:06.137239 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:06.137220 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8181c44-35b6-4760-9ef5-622dd4b8bc6d-kube-api-access-76j2x" (OuterVolumeSpecName: "kube-api-access-76j2x") pod "e8181c44-35b6-4760-9ef5-622dd4b8bc6d" (UID: "e8181c44-35b6-4760-9ef5-622dd4b8bc6d"). InnerVolumeSpecName "kube-api-access-76j2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:09:06.235846 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:06.235798 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76j2x\" (UniqueName: \"kubernetes.io/projected/e8181c44-35b6-4760-9ef5-622dd4b8bc6d-kube-api-access-76j2x\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:09:06.235846 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:06.235842 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8181c44-35b6-4760-9ef5-622dd4b8bc6d-extensions-socket-volume\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:09:06.677590 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:06.677557 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc0d3e9e-c567-4757-93a0-0760bd9e8a41" path="/var/lib/kubelet/pods/dc0d3e9e-c567-4757-93a0-0760bd9e8a41/volumes" Sep 08 10:09:06.677940 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:06.677927 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8181c44-35b6-4760-9ef5-622dd4b8bc6d" path="/var/lib/kubelet/pods/e8181c44-35b6-4760-9ef5-622dd4b8bc6d/volumes" Sep 08 10:09:06.761914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:06.761879 2576 generic.go:356] "Generic (PLEG): container finished" podID="e8181c44-35b6-4760-9ef5-622dd4b8bc6d" containerID="85042b39f2b9a9848d8d83b014eea0b000618ab9b951faf9cfea8813803c3a00" exitCode=0 Sep 08 10:09:06.762074 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:06.761926 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m" Sep 08 10:09:06.762074 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:06.761968 2576 scope.go:117] "RemoveContainer" containerID="85042b39f2b9a9848d8d83b014eea0b000618ab9b951faf9cfea8813803c3a00" Sep 08 10:09:06.766023 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:06.765990 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-v7p4m\" is forbidden: User \"system:node:ip-10-0-132-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-206.ec2.internal' and this object" podUID="e8181c44-35b6-4760-9ef5-622dd4b8bc6d" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-v7p4m" Sep 08 10:09:06.772452 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:06.772375 2576 scope.go:117] "RemoveContainer" containerID="85042b39f2b9a9848d8d83b014eea0b000618ab9b951faf9cfea8813803c3a00" Sep 08 10:09:06.779063 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:06.773541 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85042b39f2b9a9848d8d83b014eea0b000618ab9b951faf9cfea8813803c3a00\": container with ID starting with 85042b39f2b9a9848d8d83b014eea0b000618ab9b951faf9cfea8813803c3a00 not found: ID does not exist" containerID="85042b39f2b9a9848d8d83b014eea0b000618ab9b951faf9cfea8813803c3a00" Sep 08 10:09:06.779063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:06.773594 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85042b39f2b9a9848d8d83b014eea0b000618ab9b951faf9cfea8813803c3a00"} err="failed to get container status \"85042b39f2b9a9848d8d83b014eea0b000618ab9b951faf9cfea8813803c3a00\": rpc error: code = NotFound desc = could not find container \"85042b39f2b9a9848d8d83b014eea0b000618ab9b951faf9cfea8813803c3a00\": container with ID starting with 85042b39f2b9a9848d8d83b014eea0b000618ab9b951faf9cfea8813803c3a00 not found: ID does not exist" Sep 08 10:09:10.260902 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:10.260868 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:10.361867 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:10.361833 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:10.674348 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:10.674312 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:10.815151 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:10.815118 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:13.913308 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:13.913275 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:14.014955 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:14.014892 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:15.623173 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:15.623138 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:15.705289 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:15.705260 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:16.764049 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:16.764020 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" Sep 08 10:09:16.764430 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:16.764087 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" Sep 08 10:09:16.859667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:16.859636 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm"] Sep 08 10:09:16.859838 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:16.859817 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" podUID="537ac7d4-f6b5-4550-899a-f49f9a411885" containerName="manager" containerID="cri-o://215f0729d540b1dba2c274324e37f541b42b2c9ffc708f19e64fbaa69b5a8f6d" gracePeriod=10 Sep 08 10:09:17.108701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.108677 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" Sep 08 10:09:17.124934 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.124903 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/537ac7d4-f6b5-4550-899a-f49f9a411885-extensions-socket-volume\") pod \"537ac7d4-f6b5-4550-899a-f49f9a411885\" (UID: \"537ac7d4-f6b5-4550-899a-f49f9a411885\") " Sep 08 10:09:17.125065 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.124965 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vrw8\" (UniqueName: \"kubernetes.io/projected/537ac7d4-f6b5-4550-899a-f49f9a411885-kube-api-access-5vrw8\") pod \"537ac7d4-f6b5-4550-899a-f49f9a411885\" (UID: \"537ac7d4-f6b5-4550-899a-f49f9a411885\") " Sep 08 10:09:17.125315 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.125287 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537ac7d4-f6b5-4550-899a-f49f9a411885-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "537ac7d4-f6b5-4550-899a-f49f9a411885" (UID: "537ac7d4-f6b5-4550-899a-f49f9a411885"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:09:17.126998 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.126974 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537ac7d4-f6b5-4550-899a-f49f9a411885-kube-api-access-5vrw8" (OuterVolumeSpecName: "kube-api-access-5vrw8") pod "537ac7d4-f6b5-4550-899a-f49f9a411885" (UID: "537ac7d4-f6b5-4550-899a-f49f9a411885"). InnerVolumeSpecName "kube-api-access-5vrw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:09:17.226106 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.226065 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/537ac7d4-f6b5-4550-899a-f49f9a411885-extensions-socket-volume\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:09:17.226106 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.226099 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vrw8\" (UniqueName: \"kubernetes.io/projected/537ac7d4-f6b5-4550-899a-f49f9a411885-kube-api-access-5vrw8\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:09:17.308161 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.308077 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd"] Sep 08 10:09:17.308454 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.308442 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="537ac7d4-f6b5-4550-899a-f49f9a411885" containerName="manager" Sep 08 10:09:17.308495 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.308457 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="537ac7d4-f6b5-4550-899a-f49f9a411885" containerName="manager" Sep 08 10:09:17.308495 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.308465 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc0d3e9e-c567-4757-93a0-0760bd9e8a41" containerName="manager" Sep 08 10:09:17.308495 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.308471 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc0d3e9e-c567-4757-93a0-0760bd9e8a41" containerName="manager" Sep 08 10:09:17.308581 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.308536 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc0d3e9e-c567-4757-93a0-0760bd9e8a41" containerName="manager" Sep 08 10:09:17.308581 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.308545 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="537ac7d4-f6b5-4550-899a-f49f9a411885" containerName="manager" Sep 08 10:09:17.311919 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.311902 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" Sep 08 10:09:17.327028 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.326999 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjbwr\" (UniqueName: \"kubernetes.io/projected/53720794-3846-4a9f-8659-9498a91d7866-kube-api-access-cjbwr\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-twmwd\" (UID: \"53720794-3846-4a9f-8659-9498a91d7866\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" Sep 08 10:09:17.327160 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.327091 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53720794-3846-4a9f-8659-9498a91d7866-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-twmwd\" (UID: \"53720794-3846-4a9f-8659-9498a91d7866\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" Sep 08 10:09:17.336762 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.336738 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd"] Sep 08 10:09:17.428012 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.427971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53720794-3846-4a9f-8659-9498a91d7866-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-twmwd\" (UID: \"53720794-3846-4a9f-8659-9498a91d7866\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" Sep 08 10:09:17.428164 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.428049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjbwr\" (UniqueName: \"kubernetes.io/projected/53720794-3846-4a9f-8659-9498a91d7866-kube-api-access-cjbwr\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-twmwd\" (UID: \"53720794-3846-4a9f-8659-9498a91d7866\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" Sep 08 10:09:17.428376 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.428358 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53720794-3846-4a9f-8659-9498a91d7866-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-twmwd\" (UID: \"53720794-3846-4a9f-8659-9498a91d7866\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" Sep 08 10:09:17.444596 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.444557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjbwr\" (UniqueName: \"kubernetes.io/projected/53720794-3846-4a9f-8659-9498a91d7866-kube-api-access-cjbwr\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-twmwd\" (UID: \"53720794-3846-4a9f-8659-9498a91d7866\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" Sep 08 10:09:17.622166 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.622076 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" Sep 08 10:09:17.824051 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.824016 2576 generic.go:356] "Generic (PLEG): container finished" podID="537ac7d4-f6b5-4550-899a-f49f9a411885" containerID="215f0729d540b1dba2c274324e37f541b42b2c9ffc708f19e64fbaa69b5a8f6d" exitCode=0 Sep 08 10:09:17.824435 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.824077 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" Sep 08 10:09:17.824435 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.824111 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" event={"ID":"537ac7d4-f6b5-4550-899a-f49f9a411885","Type":"ContainerDied","Data":"215f0729d540b1dba2c274324e37f541b42b2c9ffc708f19e64fbaa69b5a8f6d"} Sep 08 10:09:17.824435 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.824157 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm" event={"ID":"537ac7d4-f6b5-4550-899a-f49f9a411885","Type":"ContainerDied","Data":"974aa42d2b5dd232ad558d8b654900dabaafde473b9eb7d01f41e3eb49d2e296"} Sep 08 10:09:17.824435 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.824179 2576 scope.go:117] "RemoveContainer" containerID="215f0729d540b1dba2c274324e37f541b42b2c9ffc708f19e64fbaa69b5a8f6d" Sep 08 10:09:17.833169 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.833149 2576 scope.go:117] "RemoveContainer" containerID="215f0729d540b1dba2c274324e37f541b42b2c9ffc708f19e64fbaa69b5a8f6d" Sep 08 10:09:17.833459 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:17.833438 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215f0729d540b1dba2c274324e37f541b42b2c9ffc708f19e64fbaa69b5a8f6d\": container with ID starting with 215f0729d540b1dba2c274324e37f541b42b2c9ffc708f19e64fbaa69b5a8f6d not found: ID does not exist" containerID="215f0729d540b1dba2c274324e37f541b42b2c9ffc708f19e64fbaa69b5a8f6d" Sep 08 10:09:17.833500 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.833467 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215f0729d540b1dba2c274324e37f541b42b2c9ffc708f19e64fbaa69b5a8f6d"} err="failed to get container status \"215f0729d540b1dba2c274324e37f541b42b2c9ffc708f19e64fbaa69b5a8f6d\": rpc error: code = NotFound desc = could not find container \"215f0729d540b1dba2c274324e37f541b42b2c9ffc708f19e64fbaa69b5a8f6d\": container with ID starting with 215f0729d540b1dba2c274324e37f541b42b2c9ffc708f19e64fbaa69b5a8f6d not found: ID does not exist" Sep 08 10:09:17.855072 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.855047 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm"] Sep 08 10:09:17.859725 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.859702 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vd2rm"] Sep 08 10:09:17.928967 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:17.928943 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd"] Sep 08 10:09:17.931076 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:09:17.931051 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53720794_3846_4a9f_8659_9498a91d7866.slice/crio-598dc8e772f33415526d53a2d5625d3b704e2e06c15663755d348a071dbae8d5 WatchSource:0}: Error finding container 598dc8e772f33415526d53a2d5625d3b704e2e06c15663755d348a071dbae8d5: Status 404 returned error can't find the container with id 598dc8e772f33415526d53a2d5625d3b704e2e06c15663755d348a071dbae8d5 Sep 08 10:09:18.677471 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:18.677436 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="537ac7d4-f6b5-4550-899a-f49f9a411885" path="/var/lib/kubelet/pods/537ac7d4-f6b5-4550-899a-f49f9a411885/volumes" Sep 08 10:09:18.832496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:18.832463 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" event={"ID":"53720794-3846-4a9f-8659-9498a91d7866","Type":"ContainerStarted","Data":"fccf1e181b67b7a81fc668edfe3b66460e129f3723e455b7a9f3182bcfa97928"} Sep 08 10:09:18.832857 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:18.832521 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" event={"ID":"53720794-3846-4a9f-8659-9498a91d7866","Type":"ContainerStarted","Data":"598dc8e772f33415526d53a2d5625d3b704e2e06c15663755d348a071dbae8d5"} Sep 08 10:09:18.832857 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:18.832550 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" Sep 08 10:09:18.860249 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:18.860194 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" podStartSLOduration=1.860176987 podStartE2EDuration="1.860176987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:09:18.859609501 +0000 UTC m=+468.648661304" watchObservedRunningTime="2026-09-08 10:09:18.860176987 +0000 UTC m=+468.649228791" Sep 08 10:09:22.113589 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:22.113553 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:22.212739 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:22.212705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:22.385808 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:22.385724 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:25.309028 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:25.308993 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:25.377930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:25.377900 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:29.173804 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:29.173771 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:29.246500 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:29.246457 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:29.841179 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:29.841152 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" Sep 08 10:09:29.900068 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:29.900031 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh"] Sep 08 10:09:29.900264 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:29.900240 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" podUID="47ad138f-2aa6-4d4e-b471-1fd88c995b64" containerName="manager" containerID="cri-o://6961eeeb23354dda5deeda9d375aea183fa5e423df6d1de271a47b453e220470" gracePeriod=10 Sep 08 10:09:30.155920 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.155894 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" Sep 08 10:09:30.235497 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.235463 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:30.242569 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.242543 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47ad138f-2aa6-4d4e-b471-1fd88c995b64-extensions-socket-volume\") pod \"47ad138f-2aa6-4d4e-b471-1fd88c995b64\" (UID: \"47ad138f-2aa6-4d4e-b471-1fd88c995b64\") " Sep 08 10:09:30.242731 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.242613 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfxgl\" (UniqueName: \"kubernetes.io/projected/47ad138f-2aa6-4d4e-b471-1fd88c995b64-kube-api-access-tfxgl\") pod \"47ad138f-2aa6-4d4e-b471-1fd88c995b64\" (UID: \"47ad138f-2aa6-4d4e-b471-1fd88c995b64\") " Sep 08 10:09:30.242986 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.242961 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ad138f-2aa6-4d4e-b471-1fd88c995b64-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "47ad138f-2aa6-4d4e-b471-1fd88c995b64" (UID: "47ad138f-2aa6-4d4e-b471-1fd88c995b64"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:09:30.244592 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.244569 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ad138f-2aa6-4d4e-b471-1fd88c995b64-kube-api-access-tfxgl" (OuterVolumeSpecName: "kube-api-access-tfxgl") pod "47ad138f-2aa6-4d4e-b471-1fd88c995b64" (UID: "47ad138f-2aa6-4d4e-b471-1fd88c995b64"). InnerVolumeSpecName "kube-api-access-tfxgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:09:30.343880 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.343844 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfxgl\" (UniqueName: \"kubernetes.io/projected/47ad138f-2aa6-4d4e-b471-1fd88c995b64-kube-api-access-tfxgl\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:09:30.343880 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.343872 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47ad138f-2aa6-4d4e-b471-1fd88c995b64-extensions-socket-volume\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:09:30.384186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.384097 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:30.899546 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.899514 2576 generic.go:356] "Generic (PLEG): container finished" podID="47ad138f-2aa6-4d4e-b471-1fd88c995b64" containerID="6961eeeb23354dda5deeda9d375aea183fa5e423df6d1de271a47b453e220470" exitCode=0 Sep 08 10:09:30.899677 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.899563 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" Sep 08 10:09:30.899677 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.899626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" event={"ID":"47ad138f-2aa6-4d4e-b471-1fd88c995b64","Type":"ContainerDied","Data":"6961eeeb23354dda5deeda9d375aea183fa5e423df6d1de271a47b453e220470"} Sep 08 10:09:30.899774 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.899694 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh" event={"ID":"47ad138f-2aa6-4d4e-b471-1fd88c995b64","Type":"ContainerDied","Data":"62c821869287fe4c378afb2fda856f2fca3fd4566cb7dce86644967beda24494"} Sep 08 10:09:30.899774 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.899725 2576 scope.go:117] "RemoveContainer" containerID="6961eeeb23354dda5deeda9d375aea183fa5e423df6d1de271a47b453e220470" Sep 08 10:09:30.909761 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.909686 2576 scope.go:117] "RemoveContainer" containerID="6961eeeb23354dda5deeda9d375aea183fa5e423df6d1de271a47b453e220470" Sep 08 10:09:30.910012 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:09:30.909988 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6961eeeb23354dda5deeda9d375aea183fa5e423df6d1de271a47b453e220470\": container with ID starting with 6961eeeb23354dda5deeda9d375aea183fa5e423df6d1de271a47b453e220470 not found: ID does not exist" containerID="6961eeeb23354dda5deeda9d375aea183fa5e423df6d1de271a47b453e220470" Sep 08 10:09:30.910064 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.910030 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6961eeeb23354dda5deeda9d375aea183fa5e423df6d1de271a47b453e220470"} err="failed to get container status \"6961eeeb23354dda5deeda9d375aea183fa5e423df6d1de271a47b453e220470\": rpc error: code = NotFound desc = could not find container \"6961eeeb23354dda5deeda9d375aea183fa5e423df6d1de271a47b453e220470\": container with ID starting with 6961eeeb23354dda5deeda9d375aea183fa5e423df6d1de271a47b453e220470 not found: ID does not exist" Sep 08 10:09:30.925855 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.925834 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh"] Sep 08 10:09:30.938650 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:30.938590 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rfgvh"] Sep 08 10:09:32.677838 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:32.677802 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47ad138f-2aa6-4d4e-b471-1fd88c995b64" path="/var/lib/kubelet/pods/47ad138f-2aa6-4d4e-b471-1fd88c995b64/volumes" Sep 08 10:09:32.947040 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:32.946954 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:33.048741 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.048703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:33.427138 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.427104 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t"] Sep 08 10:09:33.427464 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.427452 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ad138f-2aa6-4d4e-b471-1fd88c995b64" containerName="manager" Sep 08 10:09:33.427529 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.427466 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ad138f-2aa6-4d4e-b471-1fd88c995b64" containerName="manager" Sep 08 10:09:33.427529 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.427523 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="47ad138f-2aa6-4d4e-b471-1fd88c995b64" containerName="manager" Sep 08 10:09:33.447410 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.447381 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t"] Sep 08 10:09:33.447557 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.447490 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.449833 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.449811 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 10:09:33.449965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.449832 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c5vgb\"" Sep 08 10:09:33.463083 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.463050 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:33.572467 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.572432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48567896-0f21-404c-863a-1ece4631e205-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.572665 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.572477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48567896-0f21-404c-863a-1ece4631e205-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.572665 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.572503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48567896-0f21-404c-863a-1ece4631e205-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.572665 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.572570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48567896-0f21-404c-863a-1ece4631e205-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.572813 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.572681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48567896-0f21-404c-863a-1ece4631e205-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.572813 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.572711 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bms\" (UniqueName: \"kubernetes.io/projected/48567896-0f21-404c-863a-1ece4631e205-kube-api-access-n4bms\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.572813 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.572752 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48567896-0f21-404c-863a-1ece4631e205-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.572813 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.572773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48567896-0f21-404c-863a-1ece4631e205-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.572959 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.572834 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48567896-0f21-404c-863a-1ece4631e205-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.674107 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.674072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48567896-0f21-404c-863a-1ece4631e205-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.674302 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.674122 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48567896-0f21-404c-863a-1ece4631e205-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.674302 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.674156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48567896-0f21-404c-863a-1ece4631e205-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.674302 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.674191 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48567896-0f21-404c-863a-1ece4631e205-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.674302 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.674221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bms\" (UniqueName: \"kubernetes.io/projected/48567896-0f21-404c-863a-1ece4631e205-kube-api-access-n4bms\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.674302 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.674253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48567896-0f21-404c-863a-1ece4631e205-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.674302 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.674295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48567896-0f21-404c-863a-1ece4631e205-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.674661 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.674367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48567896-0f21-404c-863a-1ece4631e205-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.674661 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.674500 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48567896-0f21-404c-863a-1ece4631e205-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.674783 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.674662 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48567896-0f21-404c-863a-1ece4631e205-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.674840 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.674820 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48567896-0f21-404c-863a-1ece4631e205-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.674895 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.674864 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48567896-0f21-404c-863a-1ece4631e205-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.674953 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.674929 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48567896-0f21-404c-863a-1ece4631e205-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.675204 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.675133 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48567896-0f21-404c-863a-1ece4631e205-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.676509 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.676488 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48567896-0f21-404c-863a-1ece4631e205-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.676629 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.676598 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48567896-0f21-404c-863a-1ece4631e205-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.682397 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.682343 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48567896-0f21-404c-863a-1ece4631e205-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.682692 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.682415 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bms\" (UniqueName: \"kubernetes.io/projected/48567896-0f21-404c-863a-1ece4631e205-kube-api-access-n4bms\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2m56t\" (UID: \"48567896-0f21-404c-863a-1ece4631e205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.758528 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.758491 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:33.820411 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:33.820324 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:34.065813 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:34.065788 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t"] Sep 08 10:09:34.066733 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:09:34.066707 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48567896_0f21_404c_863a_1ece4631e205.slice/crio-a8bdba9d36e065bae1e82c76821024ec103f9fa5d2880a9ae6b6e6c394448b33 WatchSource:0}: Error finding container a8bdba9d36e065bae1e82c76821024ec103f9fa5d2880a9ae6b6e6c394448b33: Status 404 returned error can't find the container with id a8bdba9d36e065bae1e82c76821024ec103f9fa5d2880a9ae6b6e6c394448b33 Sep 08 10:09:34.922304 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:34.922268 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" event={"ID":"48567896-0f21-404c-863a-1ece4631e205","Type":"ContainerStarted","Data":"a8bdba9d36e065bae1e82c76821024ec103f9fa5d2880a9ae6b6e6c394448b33"} Sep 08 10:09:35.954490 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:35.954455 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:36.968516 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:36.968481 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:37.280677 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:37.280636 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:09:37.280806 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:37.280724 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:09:37.280806 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:37.280766 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:09:37.940047 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:37.940010 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" event={"ID":"48567896-0f21-404c-863a-1ece4631e205","Type":"ContainerStarted","Data":"b05ad3d4161e7207f970598ae7abd1207c1948fed24f0eca4e0a66dc48f8a6d6"} Sep 08 10:09:38.759244 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:38.759205 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:38.763755 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:38.763730 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:38.800518 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:38.800469 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" podStartSLOduration=2.589042444 podStartE2EDuration="5.800454867s" totalImagesPullingTime="3.211412423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:09:37.974536179 +0000 UTC m=+487.763587983" watchObservedRunningTime="2026-09-08 10:09:38.800454867 +0000 UTC m=+488.589506670" Sep 08 10:09:38.945555 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:38.945521 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:38.946483 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:38.946459 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:09:41.763230 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:41.763195 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:41.863513 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:41.863477 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:41.963443 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:41.963415 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:45.040061 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.040025 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:45.041043 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.041022 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-6s69r"] Sep 08 10:09:45.044633 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.044603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" Sep 08 10:09:45.046953 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.046934 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 08 10:09:45.053820 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.053794 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-6s69r"] Sep 08 10:09:45.138774 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.138742 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-6s69r"] Sep 08 10:09:45.146320 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.146292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:45.171583 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.171556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mjgz\" (UniqueName: \"kubernetes.io/projected/6265c99c-1852-4d11-997a-165a64579e89-kube-api-access-6mjgz\") pod \"limitador-limitador-859c4d96d6-6s69r\" (UID: \"6265c99c-1852-4d11-997a-165a64579e89\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" Sep 08 10:09:45.171732 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.171716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6265c99c-1852-4d11-997a-165a64579e89-config-file\") pod \"limitador-limitador-859c4d96d6-6s69r\" (UID: \"6265c99c-1852-4d11-997a-165a64579e89\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" Sep 08 10:09:45.207466 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.207442 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:45.272611 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.272575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjgz\" (UniqueName: \"kubernetes.io/projected/6265c99c-1852-4d11-997a-165a64579e89-kube-api-access-6mjgz\") pod \"limitador-limitador-859c4d96d6-6s69r\" (UID: \"6265c99c-1852-4d11-997a-165a64579e89\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" Sep 08 10:09:45.272804 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.272732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6265c99c-1852-4d11-997a-165a64579e89-config-file\") pod \"limitador-limitador-859c4d96d6-6s69r\" (UID: \"6265c99c-1852-4d11-997a-165a64579e89\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" Sep 08 10:09:45.273320 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.273302 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6265c99c-1852-4d11-997a-165a64579e89-config-file\") pod \"limitador-limitador-859c4d96d6-6s69r\" (UID: \"6265c99c-1852-4d11-997a-165a64579e89\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" Sep 08 10:09:45.283986 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.283959 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mjgz\" (UniqueName: \"kubernetes.io/projected/6265c99c-1852-4d11-997a-165a64579e89-kube-api-access-6mjgz\") pod \"limitador-limitador-859c4d96d6-6s69r\" (UID: \"6265c99c-1852-4d11-997a-165a64579e89\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" Sep 08 10:09:45.354974 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.354900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" Sep 08 10:09:45.498185 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.498148 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:09:45.502693 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.502672 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-s5tht" Sep 08 10:09:45.519360 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.519335 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:09:45.565385 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.565350 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:09:45.576343 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.576314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db58c\" (UniqueName: \"kubernetes.io/projected/a27e588e-b522-447f-a566-2c8320de409e-kube-api-access-db58c\") pod \"limitador-limitador-85d9db4d5d-s5tht\" (UID: \"a27e588e-b522-447f-a566-2c8320de409e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-s5tht" Sep 08 10:09:45.576466 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.576414 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a27e588e-b522-447f-a566-2c8320de409e-config-file\") pod \"limitador-limitador-85d9db4d5d-s5tht\" (UID: \"a27e588e-b522-447f-a566-2c8320de409e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-s5tht" Sep 08 10:09:45.629182 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.629155 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-6s69r"] Sep 08 10:09:45.630188 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:09:45.630155 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6265c99c_1852_4d11_997a_165a64579e89.slice/crio-6b8749136c73cdc38c59e3229e26fba0dc0375d0ba5acebc0f29a3202a3e5037 WatchSource:0}: Error finding container 6b8749136c73cdc38c59e3229e26fba0dc0375d0ba5acebc0f29a3202a3e5037: Status 404 returned error can't find the container with id 6b8749136c73cdc38c59e3229e26fba0dc0375d0ba5acebc0f29a3202a3e5037 Sep 08 10:09:45.677044 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.677015 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a27e588e-b522-447f-a566-2c8320de409e-config-file\") pod \"limitador-limitador-85d9db4d5d-s5tht\" (UID: \"a27e588e-b522-447f-a566-2c8320de409e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-s5tht" Sep 08 10:09:45.677157 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.677070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db58c\" (UniqueName: \"kubernetes.io/projected/a27e588e-b522-447f-a566-2c8320de409e-kube-api-access-db58c\") pod \"limitador-limitador-85d9db4d5d-s5tht\" (UID: \"a27e588e-b522-447f-a566-2c8320de409e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-s5tht" Sep 08 10:09:45.677698 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.677677 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a27e588e-b522-447f-a566-2c8320de409e-config-file\") pod \"limitador-limitador-85d9db4d5d-s5tht\" (UID: \"a27e588e-b522-447f-a566-2c8320de409e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-s5tht" Sep 08 10:09:45.686122 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.686100 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db58c\" (UniqueName: \"kubernetes.io/projected/a27e588e-b522-447f-a566-2c8320de409e-kube-api-access-db58c\") pod \"limitador-limitador-85d9db4d5d-s5tht\" (UID: \"a27e588e-b522-447f-a566-2c8320de409e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-s5tht" Sep 08 10:09:45.812973 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.812935 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-s5tht" Sep 08 10:09:45.982214 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:45.982180 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" event={"ID":"6265c99c-1852-4d11-997a-165a64579e89","Type":"ContainerStarted","Data":"6b8749136c73cdc38c59e3229e26fba0dc0375d0ba5acebc0f29a3202a3e5037"} Sep 08 10:09:46.051910 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:46.051879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:46.091335 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:46.091305 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:09:46.092037 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:09:46.092015 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27e588e_b522_447f_a566_2c8320de409e.slice/crio-6da1ece255e0e169f6f021a6d6ff5e9e7a5aa52b0f40360a87e087ad6cf4e813 WatchSource:0}: Error finding container 6da1ece255e0e169f6f021a6d6ff5e9e7a5aa52b0f40360a87e087ad6cf4e813: Status 404 returned error can't find the container with id 6da1ece255e0e169f6f021a6d6ff5e9e7a5aa52b0f40360a87e087ad6cf4e813 Sep 08 10:09:46.147187 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:46.147155 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:09:46.988884 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:46.988842 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-s5tht" event={"ID":"a27e588e-b522-447f-a566-2c8320de409e","Type":"ContainerStarted","Data":"6da1ece255e0e169f6f021a6d6ff5e9e7a5aa52b0f40360a87e087ad6cf4e813"} Sep 08 10:09:48.001208 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:48.001170 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-s5tht" event={"ID":"a27e588e-b522-447f-a566-2c8320de409e","Type":"ContainerStarted","Data":"ba248d862213d0811a178725654f22340d903445a577c52fea138a77ede6a10f"} Sep 08 10:09:48.001724 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:48.001280 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-s5tht" Sep 08 10:09:51.018726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:51.018690 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" event={"ID":"6265c99c-1852-4d11-997a-165a64579e89","Type":"ContainerStarted","Data":"b6525f84dc9d14865cc0668a4bc4605324ff0d30cfbc79ac472bee47fadbd00c"} Sep 08 10:09:51.019173 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:51.018811 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" Sep 08 10:09:51.043674 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:51.043609 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" podStartSLOduration=1.525001474 podStartE2EDuration="6.043592337s" totalImagesPullingTime="4.518590863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:09:51.041711514 +0000 UTC m=+500.830763318" watchObservedRunningTime="2026-09-08 10:09:51.043592337 +0000 UTC m=+500.832644198" Sep 08 10:09:51.043838 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:51.043698 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-s5tht" podStartSLOduration=4.780125394 podStartE2EDuration="6.043692207s" totalImagesPullingTime="1.263566813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:09:48.046611857 +0000 UTC m=+497.835663660" watchObservedRunningTime="2026-09-08 10:09:51.043692207 +0000 UTC m=+500.832744011" Sep 08 10:09:59.007868 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:59.007834 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-s5tht" Sep 08 10:09:59.107966 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:59.107931 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-6s69r"] Sep 08 10:09:59.108150 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:59.108124 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" podUID="6265c99c-1852-4d11-997a-165a64579e89" containerName="limitador" containerID="cri-o://b6525f84dc9d14865cc0668a4bc4605324ff0d30cfbc79ac472bee47fadbd00c" gracePeriod=30 Sep 08 10:09:59.108899 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:59.108741 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" Sep 08 10:09:59.652999 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:59.652976 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" Sep 08 10:09:59.793301 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:59.793217 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mjgz\" (UniqueName: \"kubernetes.io/projected/6265c99c-1852-4d11-997a-165a64579e89-kube-api-access-6mjgz\") pod \"6265c99c-1852-4d11-997a-165a64579e89\" (UID: \"6265c99c-1852-4d11-997a-165a64579e89\") " Sep 08 10:09:59.793455 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:59.793303 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6265c99c-1852-4d11-997a-165a64579e89-config-file\") pod \"6265c99c-1852-4d11-997a-165a64579e89\" (UID: \"6265c99c-1852-4d11-997a-165a64579e89\") " Sep 08 10:09:59.793718 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:59.793688 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6265c99c-1852-4d11-997a-165a64579e89-config-file" (OuterVolumeSpecName: "config-file") pod "6265c99c-1852-4d11-997a-165a64579e89" (UID: "6265c99c-1852-4d11-997a-165a64579e89"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:09:59.795406 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:59.795382 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6265c99c-1852-4d11-997a-165a64579e89-kube-api-access-6mjgz" (OuterVolumeSpecName: "kube-api-access-6mjgz") pod "6265c99c-1852-4d11-997a-165a64579e89" (UID: "6265c99c-1852-4d11-997a-165a64579e89"). InnerVolumeSpecName "kube-api-access-6mjgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:09:59.894442 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:59.894403 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mjgz\" (UniqueName: \"kubernetes.io/projected/6265c99c-1852-4d11-997a-165a64579e89-kube-api-access-6mjgz\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:09:59.894442 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:09:59.894434 2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6265c99c-1852-4d11-997a-165a64579e89-config-file\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:10:00.069426 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:00.069333 2576 generic.go:356] "Generic (PLEG): container finished" podID="6265c99c-1852-4d11-997a-165a64579e89" containerID="b6525f84dc9d14865cc0668a4bc4605324ff0d30cfbc79ac472bee47fadbd00c" exitCode=0 Sep 08 10:10:00.069426 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:00.069382 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" Sep 08 10:10:00.069426 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:00.069403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" event={"ID":"6265c99c-1852-4d11-997a-165a64579e89","Type":"ContainerDied","Data":"b6525f84dc9d14865cc0668a4bc4605324ff0d30cfbc79ac472bee47fadbd00c"} Sep 08 10:10:00.069946 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:00.069459 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-6s69r" event={"ID":"6265c99c-1852-4d11-997a-165a64579e89","Type":"ContainerDied","Data":"6b8749136c73cdc38c59e3229e26fba0dc0375d0ba5acebc0f29a3202a3e5037"} Sep 08 10:10:00.069946 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:00.069488 2576 scope.go:117] "RemoveContainer" containerID="b6525f84dc9d14865cc0668a4bc4605324ff0d30cfbc79ac472bee47fadbd00c" Sep 08 10:10:00.079200 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:00.079181 2576 scope.go:117] "RemoveContainer" containerID="b6525f84dc9d14865cc0668a4bc4605324ff0d30cfbc79ac472bee47fadbd00c" Sep 08 10:10:00.079493 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:10:00.079449 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6525f84dc9d14865cc0668a4bc4605324ff0d30cfbc79ac472bee47fadbd00c\": container with ID starting with b6525f84dc9d14865cc0668a4bc4605324ff0d30cfbc79ac472bee47fadbd00c not found: ID does not exist" containerID="b6525f84dc9d14865cc0668a4bc4605324ff0d30cfbc79ac472bee47fadbd00c" Sep 08 10:10:00.079493 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:00.079475 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6525f84dc9d14865cc0668a4bc4605324ff0d30cfbc79ac472bee47fadbd00c"} err="failed to get container status \"b6525f84dc9d14865cc0668a4bc4605324ff0d30cfbc79ac472bee47fadbd00c\": rpc error: code = NotFound desc = could not find container \"b6525f84dc9d14865cc0668a4bc4605324ff0d30cfbc79ac472bee47fadbd00c\": container with ID starting with b6525f84dc9d14865cc0668a4bc4605324ff0d30cfbc79ac472bee47fadbd00c not found: ID does not exist" Sep 08 10:10:00.094181 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:00.094159 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-6s69r"] Sep 08 10:10:00.098005 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:00.097984 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-6s69r"] Sep 08 10:10:00.682255 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:00.682218 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6265c99c-1852-4d11-997a-165a64579e89" path="/var/lib/kubelet/pods/6265c99c-1852-4d11-997a-165a64579e89/volumes" Sep 08 10:10:00.879558 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:00.879522 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:00.978499 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:00.978409 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:01.101812 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:01.101776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:11.229640 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:11.229592 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:14.304158 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:14.304124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:14.464457 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:14.464419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:21.299984 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.299950 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:21.334240 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.334203 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bbdd8d44-8xxxt"] Sep 08 10:10:21.334583 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.334569 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6265c99c-1852-4d11-997a-165a64579e89" containerName="limitador" Sep 08 10:10:21.334650 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.334587 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6265c99c-1852-4d11-997a-165a64579e89" containerName="limitador" Sep 08 10:10:21.334694 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.334689 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6265c99c-1852-4d11-997a-165a64579e89" containerName="limitador" Sep 08 10:10:21.342489 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.342460 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:21.345395 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.345370 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 10:10:21.345746 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.345377 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nf7fl\"" Sep 08 10:10:21.346018 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.345473 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 10:10:21.346129 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.345510 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 10:10:21.347515 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.347452 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbdd8d44-8xxxt"] Sep 08 10:10:21.374430 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.374397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:21.388033 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.388007 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-webhook-cert\") pod \"maas-controller-7bbdd8d44-8xxxt\" (UID: \"0db1c035-b132-40ad-b875-54e19a1a0c6e\") " pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:21.388163 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.388061 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvhps\" (UniqueName: \"kubernetes.io/projected/0db1c035-b132-40ad-b875-54e19a1a0c6e-kube-api-access-mvhps\") pod \"maas-controller-7bbdd8d44-8xxxt\" (UID: \"0db1c035-b132-40ad-b875-54e19a1a0c6e\") " pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:21.388163 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.388098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-metrics-certs\") pod \"maas-controller-7bbdd8d44-8xxxt\" (UID: \"0db1c035-b132-40ad-b875-54e19a1a0c6e\") " pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:21.437079 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.437048 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:21.489294 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.489262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-webhook-cert\") pod \"maas-controller-7bbdd8d44-8xxxt\" (UID: \"0db1c035-b132-40ad-b875-54e19a1a0c6e\") " pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:21.489450 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.489305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvhps\" (UniqueName: \"kubernetes.io/projected/0db1c035-b132-40ad-b875-54e19a1a0c6e-kube-api-access-mvhps\") pod \"maas-controller-7bbdd8d44-8xxxt\" (UID: \"0db1c035-b132-40ad-b875-54e19a1a0c6e\") " pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:21.489450 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.489330 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-metrics-certs\") pod \"maas-controller-7bbdd8d44-8xxxt\" (UID: \"0db1c035-b132-40ad-b875-54e19a1a0c6e\") " pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:21.489450 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:10:21.489417 2576 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 08 10:10:21.489450 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:10:21.489423 2576 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 08 10:10:21.489666 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:10:21.489482 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-metrics-certs podName:0db1c035-b132-40ad-b875-54e19a1a0c6e nodeName:}" failed. No retries permitted until 2026-09-08 10:10:21.989467335 +0000 UTC m=+531.778519129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-metrics-certs") pod "maas-controller-7bbdd8d44-8xxxt" (UID: "0db1c035-b132-40ad-b875-54e19a1a0c6e") : secret "maas-controller-metrics-tls" not found Sep 08 10:10:21.489666 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:10:21.489495 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-webhook-cert podName:0db1c035-b132-40ad-b875-54e19a1a0c6e nodeName:}" failed. No retries permitted until 2026-09-08 10:10:21.989489162 +0000 UTC m=+531.778540944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-webhook-cert") pod "maas-controller-7bbdd8d44-8xxxt" (UID: "0db1c035-b132-40ad-b875-54e19a1a0c6e") : secret "maas-controller-webhook-cert" not found Sep 08 10:10:21.500155 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.500125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvhps\" (UniqueName: \"kubernetes.io/projected/0db1c035-b132-40ad-b875-54e19a1a0c6e-kube-api-access-mvhps\") pod \"maas-controller-7bbdd8d44-8xxxt\" (UID: \"0db1c035-b132-40ad-b875-54e19a1a0c6e\") " pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:21.509939 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.509911 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-869d99c85c-sfn7r"] Sep 08 10:10:21.513832 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.513813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-869d99c85c-sfn7r" Sep 08 10:10:21.524001 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.523977 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-869d99c85c-sfn7r"] Sep 08 10:10:21.582915 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.582831 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:21.589929 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.589902 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnt68\" (UniqueName: \"kubernetes.io/projected/69e54261-e36c-45ce-99cb-d2af888add0b-kube-api-access-xnt68\") pod \"maas-controller-869d99c85c-sfn7r\" (UID: \"69e54261-e36c-45ce-99cb-d2af888add0b\") " pod="opendatahub/maas-controller-869d99c85c-sfn7r" Sep 08 10:10:21.590033 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.589950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69e54261-e36c-45ce-99cb-d2af888add0b-webhook-cert\") pod \"maas-controller-869d99c85c-sfn7r\" (UID: \"69e54261-e36c-45ce-99cb-d2af888add0b\") " pod="opendatahub/maas-controller-869d99c85c-sfn7r" Sep 08 10:10:21.590073 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.590060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69e54261-e36c-45ce-99cb-d2af888add0b-metrics-certs\") pod \"maas-controller-869d99c85c-sfn7r\" (UID: \"69e54261-e36c-45ce-99cb-d2af888add0b\") " pod="opendatahub/maas-controller-869d99c85c-sfn7r" Sep 08 10:10:21.685133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.685100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:21.691261 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.691232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnt68\" (UniqueName: \"kubernetes.io/projected/69e54261-e36c-45ce-99cb-d2af888add0b-kube-api-access-xnt68\") pod \"maas-controller-869d99c85c-sfn7r\" (UID: \"69e54261-e36c-45ce-99cb-d2af888add0b\") " pod="opendatahub/maas-controller-869d99c85c-sfn7r" Sep 08 10:10:21.691329 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.691293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69e54261-e36c-45ce-99cb-d2af888add0b-webhook-cert\") pod \"maas-controller-869d99c85c-sfn7r\" (UID: \"69e54261-e36c-45ce-99cb-d2af888add0b\") " pod="opendatahub/maas-controller-869d99c85c-sfn7r" Sep 08 10:10:21.691372 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.691356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69e54261-e36c-45ce-99cb-d2af888add0b-metrics-certs\") pod \"maas-controller-869d99c85c-sfn7r\" (UID: \"69e54261-e36c-45ce-99cb-d2af888add0b\") " pod="opendatahub/maas-controller-869d99c85c-sfn7r" Sep 08 10:10:21.693887 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.693864 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69e54261-e36c-45ce-99cb-d2af888add0b-webhook-cert\") pod \"maas-controller-869d99c85c-sfn7r\" (UID: \"69e54261-e36c-45ce-99cb-d2af888add0b\") " pod="opendatahub/maas-controller-869d99c85c-sfn7r" Sep 08 10:10:21.693990 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.693914 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69e54261-e36c-45ce-99cb-d2af888add0b-metrics-certs\") pod \"maas-controller-869d99c85c-sfn7r\" (UID: \"69e54261-e36c-45ce-99cb-d2af888add0b\") " pod="opendatahub/maas-controller-869d99c85c-sfn7r" Sep 08 10:10:21.701355 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.701328 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnt68\" (UniqueName: \"kubernetes.io/projected/69e54261-e36c-45ce-99cb-d2af888add0b-kube-api-access-xnt68\") pod \"maas-controller-869d99c85c-sfn7r\" (UID: \"69e54261-e36c-45ce-99cb-d2af888add0b\") " pod="opendatahub/maas-controller-869d99c85c-sfn7r" Sep 08 10:10:21.826115 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.826084 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-869d99c85c-sfn7r" Sep 08 10:10:21.994525 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.994487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-webhook-cert\") pod \"maas-controller-7bbdd8d44-8xxxt\" (UID: \"0db1c035-b132-40ad-b875-54e19a1a0c6e\") " pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:21.994712 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.994543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-metrics-certs\") pod \"maas-controller-7bbdd8d44-8xxxt\" (UID: \"0db1c035-b132-40ad-b875-54e19a1a0c6e\") " pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:21.997069 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.997039 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-metrics-certs\") pod \"maas-controller-7bbdd8d44-8xxxt\" (UID: \"0db1c035-b132-40ad-b875-54e19a1a0c6e\") " pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:21.997185 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:21.997039 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-webhook-cert\") pod \"maas-controller-7bbdd8d44-8xxxt\" (UID: \"0db1c035-b132-40ad-b875-54e19a1a0c6e\") " pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:22.101387 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:22.101359 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-869d99c85c-sfn7r"] Sep 08 10:10:22.102524 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:10:22.102501 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e54261_e36c_45ce_99cb_d2af888add0b.slice/crio-7c5db6dbf5b73de29b415dfd135851bb4acca264cd0289f90032aacad53e29b2 WatchSource:0}: Error finding container 7c5db6dbf5b73de29b415dfd135851bb4acca264cd0289f90032aacad53e29b2: Status 404 returned error can't find the container with id 7c5db6dbf5b73de29b415dfd135851bb4acca264cd0289f90032aacad53e29b2 Sep 08 10:10:22.147607 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:22.147577 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:22.192020 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:22.191985 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-869d99c85c-sfn7r" event={"ID":"69e54261-e36c-45ce-99cb-d2af888add0b","Type":"ContainerStarted","Data":"7c5db6dbf5b73de29b415dfd135851bb4acca264cd0289f90032aacad53e29b2"} Sep 08 10:10:22.255538 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:22.255501 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:22.527516 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:22.527494 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbdd8d44-8xxxt"] Sep 08 10:10:22.529611 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:10:22.529582 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db1c035_b132_40ad_b875_54e19a1a0c6e.slice/crio-89e0ab3b596a553d042c3bc467a348150839a73932a9f0a3b86461386969c345 WatchSource:0}: Error finding container 89e0ab3b596a553d042c3bc467a348150839a73932a9f0a3b86461386969c345: Status 404 returned error can't find the container with id 89e0ab3b596a553d042c3bc467a348150839a73932a9f0a3b86461386969c345 Sep 08 10:10:23.199499 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:23.199459 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" event={"ID":"0db1c035-b132-40ad-b875-54e19a1a0c6e","Type":"ContainerStarted","Data":"89e0ab3b596a553d042c3bc467a348150839a73932a9f0a3b86461386969c345"} Sep 08 10:10:24.576391 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:24.576348 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:24.746461 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:24.746423 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:25.213756 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:25.213717 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" event={"ID":"0db1c035-b132-40ad-b875-54e19a1a0c6e","Type":"ContainerStarted","Data":"57698a9cc3df0c54d7eda58dabcb43cc6fa2443fd0c1d30e3a5533faba5e4979"} Sep 08 10:10:25.213972 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:25.213875 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:25.215497 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:25.215472 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-869d99c85c-sfn7r" event={"ID":"69e54261-e36c-45ce-99cb-d2af888add0b","Type":"ContainerStarted","Data":"9a79419ec943e746adf5195a2ba892a872fa31f090eda54aa8622fad056957d3"} Sep 08 10:10:25.215638 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:25.215611 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-869d99c85c-sfn7r" Sep 08 10:10:25.246133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:25.246070 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" podStartSLOduration=2.39562302 podStartE2EDuration="4.246054141s" totalImagesPullingTime="1.850431121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:10:25.244859242 +0000 UTC m=+535.033911047" watchObservedRunningTime="2026-09-08 10:10:25.246054141 +0000 UTC m=+535.035105945" Sep 08 10:10:25.277629 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:25.277564 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-869d99c85c-sfn7r" podStartSLOduration=2.000307327 podStartE2EDuration="4.277546803s" totalImagesPullingTime="2.277239476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:10:25.276686756 +0000 UTC m=+535.065738561" watchObservedRunningTime="2026-09-08 10:10:25.277546803 +0000 UTC m=+535.066598606" Sep 08 10:10:36.222821 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:36.222784 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-869d99c85c-sfn7r" Sep 08 10:10:36.223247 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:36.222944 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:36.305898 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:36.305853 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbdd8d44-8xxxt"] Sep 08 10:10:36.306147 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:36.306092 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" podUID="0db1c035-b132-40ad-b875-54e19a1a0c6e" containerName="manager" containerID="cri-o://57698a9cc3df0c54d7eda58dabcb43cc6fa2443fd0c1d30e3a5533faba5e4979" gracePeriod=30 Sep 08 10:10:36.557151 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:36.557126 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:36.629759 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:36.629727 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-webhook-cert\") pod \"0db1c035-b132-40ad-b875-54e19a1a0c6e\" (UID: \"0db1c035-b132-40ad-b875-54e19a1a0c6e\") " Sep 08 10:10:36.629759 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:36.629759 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvhps\" (UniqueName: \"kubernetes.io/projected/0db1c035-b132-40ad-b875-54e19a1a0c6e-kube-api-access-mvhps\") pod \"0db1c035-b132-40ad-b875-54e19a1a0c6e\" (UID: \"0db1c035-b132-40ad-b875-54e19a1a0c6e\") " Sep 08 10:10:36.629949 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:36.629848 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-metrics-certs\") pod \"0db1c035-b132-40ad-b875-54e19a1a0c6e\" (UID: \"0db1c035-b132-40ad-b875-54e19a1a0c6e\") " Sep 08 10:10:36.631951 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:36.631920 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0db1c035-b132-40ad-b875-54e19a1a0c6e" (UID: "0db1c035-b132-40ad-b875-54e19a1a0c6e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:10:36.632092 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:36.631987 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0db1c035-b132-40ad-b875-54e19a1a0c6e" (UID: "0db1c035-b132-40ad-b875-54e19a1a0c6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:10:36.632092 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:36.632014 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db1c035-b132-40ad-b875-54e19a1a0c6e-kube-api-access-mvhps" (OuterVolumeSpecName: "kube-api-access-mvhps") pod "0db1c035-b132-40ad-b875-54e19a1a0c6e" (UID: "0db1c035-b132-40ad-b875-54e19a1a0c6e"). InnerVolumeSpecName "kube-api-access-mvhps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:10:36.730843 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:36.730775 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:10:36.730843 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:36.730797 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0db1c035-b132-40ad-b875-54e19a1a0c6e-webhook-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:10:36.730843 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:36.730807 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvhps\" (UniqueName: \"kubernetes.io/projected/0db1c035-b132-40ad-b875-54e19a1a0c6e-kube-api-access-mvhps\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:10:37.280114 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:37.280077 2576 generic.go:356] "Generic (PLEG): container finished" podID="0db1c035-b132-40ad-b875-54e19a1a0c6e" containerID="57698a9cc3df0c54d7eda58dabcb43cc6fa2443fd0c1d30e3a5533faba5e4979" exitCode=0 Sep 08 10:10:37.280562 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:37.280126 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" Sep 08 10:10:37.280562 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:37.280164 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" event={"ID":"0db1c035-b132-40ad-b875-54e19a1a0c6e","Type":"ContainerDied","Data":"57698a9cc3df0c54d7eda58dabcb43cc6fa2443fd0c1d30e3a5533faba5e4979"} Sep 08 10:10:37.280562 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:37.280222 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbdd8d44-8xxxt" event={"ID":"0db1c035-b132-40ad-b875-54e19a1a0c6e","Type":"ContainerDied","Data":"89e0ab3b596a553d042c3bc467a348150839a73932a9f0a3b86461386969c345"} Sep 08 10:10:37.280562 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:37.280251 2576 scope.go:117] "RemoveContainer" containerID="57698a9cc3df0c54d7eda58dabcb43cc6fa2443fd0c1d30e3a5533faba5e4979" Sep 08 10:10:37.289312 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:37.289292 2576 scope.go:117] "RemoveContainer" containerID="57698a9cc3df0c54d7eda58dabcb43cc6fa2443fd0c1d30e3a5533faba5e4979" Sep 08 10:10:37.289572 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:10:37.289552 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57698a9cc3df0c54d7eda58dabcb43cc6fa2443fd0c1d30e3a5533faba5e4979\": container with ID starting with 57698a9cc3df0c54d7eda58dabcb43cc6fa2443fd0c1d30e3a5533faba5e4979 not found: ID does not exist" containerID="57698a9cc3df0c54d7eda58dabcb43cc6fa2443fd0c1d30e3a5533faba5e4979" Sep 08 10:10:37.289694 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:37.289579 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57698a9cc3df0c54d7eda58dabcb43cc6fa2443fd0c1d30e3a5533faba5e4979"} err="failed to get container status \"57698a9cc3df0c54d7eda58dabcb43cc6fa2443fd0c1d30e3a5533faba5e4979\": rpc error: code = NotFound desc = could not find container \"57698a9cc3df0c54d7eda58dabcb43cc6fa2443fd0c1d30e3a5533faba5e4979\": container with ID starting with 57698a9cc3df0c54d7eda58dabcb43cc6fa2443fd0c1d30e3a5533faba5e4979 not found: ID does not exist" Sep 08 10:10:37.299449 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:37.299428 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbdd8d44-8xxxt"] Sep 08 10:10:37.303847 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:37.303824 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bbdd8d44-8xxxt"] Sep 08 10:10:38.678083 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:38.678048 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0db1c035-b132-40ad-b875-54e19a1a0c6e" path="/var/lib/kubelet/pods/0db1c035-b132-40ad-b875-54e19a1a0c6e/volumes" Sep 08 10:10:45.348099 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:45.348045 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:45.458416 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:45.458383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:56.556092 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:56.556055 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:56.661098 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:56.661063 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:58.105430 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.105394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:58.318776 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.318741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:58.334164 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.334131 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5"] Sep 08 10:10:58.334498 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.334484 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db1c035-b132-40ad-b875-54e19a1a0c6e" containerName="manager" Sep 08 10:10:58.334554 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.334500 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db1c035-b132-40ad-b875-54e19a1a0c6e" containerName="manager" Sep 08 10:10:58.334607 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.334589 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0db1c035-b132-40ad-b875-54e19a1a0c6e" containerName="manager" Sep 08 10:10:58.338947 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.338926 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:10:58.343214 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.343193 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 10:10:58.343484 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.343467 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 08 10:10:58.343817 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.343798 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 08 10:10:58.343891 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.343809 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 08 10:10:58.352523 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.352497 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5"] Sep 08 10:10:58.442665 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.442608 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:58.524769 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.524735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/743cdc55-c328-4650-b2f5-f0bdff2c89ee-maas-api-tls\") pod \"maas-api-787f6cbf7b-kvfw5\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:10:58.524950 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.524821 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/743cdc55-c328-4650-b2f5-f0bdff2c89ee-metrics-certs\") pod \"maas-api-787f6cbf7b-kvfw5\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:10:58.524950 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.524855 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/743cdc55-c328-4650-b2f5-f0bdff2c89ee-openshift-service-ca\") pod \"maas-api-787f6cbf7b-kvfw5\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:10:58.524950 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.524891 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hksr8\" (UniqueName: \"kubernetes.io/projected/743cdc55-c328-4650-b2f5-f0bdff2c89ee-kube-api-access-hksr8\") pod \"maas-api-787f6cbf7b-kvfw5\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:10:58.554388 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.554360 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:58.625671 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.625645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/743cdc55-c328-4650-b2f5-f0bdff2c89ee-maas-api-tls\") pod \"maas-api-787f6cbf7b-kvfw5\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:10:58.625834 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.625703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/743cdc55-c328-4650-b2f5-f0bdff2c89ee-metrics-certs\") pod \"maas-api-787f6cbf7b-kvfw5\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:10:58.625834 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.625727 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/743cdc55-c328-4650-b2f5-f0bdff2c89ee-openshift-service-ca\") pod \"maas-api-787f6cbf7b-kvfw5\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:10:58.625834 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.625744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hksr8\" (UniqueName: \"kubernetes.io/projected/743cdc55-c328-4650-b2f5-f0bdff2c89ee-kube-api-access-hksr8\") pod \"maas-api-787f6cbf7b-kvfw5\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:10:58.626460 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.626440 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/743cdc55-c328-4650-b2f5-f0bdff2c89ee-openshift-service-ca\") pod \"maas-api-787f6cbf7b-kvfw5\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:10:58.628079 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.628058 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/743cdc55-c328-4650-b2f5-f0bdff2c89ee-metrics-certs\") pod \"maas-api-787f6cbf7b-kvfw5\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:10:58.628165 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.628106 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/743cdc55-c328-4650-b2f5-f0bdff2c89ee-maas-api-tls\") pod \"maas-api-787f6cbf7b-kvfw5\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:10:58.640154 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.640130 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hksr8\" (UniqueName: \"kubernetes.io/projected/743cdc55-c328-4650-b2f5-f0bdff2c89ee-kube-api-access-hksr8\") pod \"maas-api-787f6cbf7b-kvfw5\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:10:58.648818 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.648801 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:10:58.936037 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:58.936012 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5"] Sep 08 10:10:58.937444 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:10:58.937416 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743cdc55_c328_4650_b2f5_f0bdff2c89ee.slice/crio-f59350fd0118daf0a1926c1916c9a75727f199cb8b4f941879789e969fbf54f0 WatchSource:0}: Error finding container f59350fd0118daf0a1926c1916c9a75727f199cb8b4f941879789e969fbf54f0: Status 404 returned error can't find the container with id f59350fd0118daf0a1926c1916c9a75727f199cb8b4f941879789e969fbf54f0 Sep 08 10:10:59.075964 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.075869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:59.277993 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.277949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:10:59.335693 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.335582 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd"] Sep 08 10:10:59.340097 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.340065 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd" Sep 08 10:10:59.343350 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.343328 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vbrdv\"" Sep 08 10:10:59.343471 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.343367 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 08 10:10:59.358998 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.358966 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:10:59.359246 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.359228 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd"] Sep 08 10:10:59.395567 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.395530 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" event={"ID":"743cdc55-c328-4650-b2f5-f0bdff2c89ee","Type":"ContainerStarted","Data":"f59350fd0118daf0a1926c1916c9a75727f199cb8b4f941879789e969fbf54f0"} Sep 08 10:10:59.432285 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.432247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6596df5-5301-45a5-9116-dc2a4268bc80-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-qljfd\" (UID: \"e6596df5-5301-45a5-9116-dc2a4268bc80\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd" Sep 08 10:10:59.432698 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.432670 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9ppm\" (UniqueName: \"kubernetes.io/projected/e6596df5-5301-45a5-9116-dc2a4268bc80-kube-api-access-q9ppm\") pod \"payload-pre-processing-59fbc47bf8-qljfd\" (UID: \"e6596df5-5301-45a5-9116-dc2a4268bc80\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd" Sep 08 10:10:59.533260 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.533226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9ppm\" (UniqueName: \"kubernetes.io/projected/e6596df5-5301-45a5-9116-dc2a4268bc80-kube-api-access-q9ppm\") pod \"payload-pre-processing-59fbc47bf8-qljfd\" (UID: \"e6596df5-5301-45a5-9116-dc2a4268bc80\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd" Sep 08 10:10:59.533401 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.533281 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6596df5-5301-45a5-9116-dc2a4268bc80-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-qljfd\" (UID: \"e6596df5-5301-45a5-9116-dc2a4268bc80\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd" Sep 08 10:10:59.534025 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.534001 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6596df5-5301-45a5-9116-dc2a4268bc80-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-qljfd\" (UID: \"e6596df5-5301-45a5-9116-dc2a4268bc80\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd" Sep 08 10:10:59.545961 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.545937 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9ppm\" (UniqueName: \"kubernetes.io/projected/e6596df5-5301-45a5-9116-dc2a4268bc80-kube-api-access-q9ppm\") pod \"payload-pre-processing-59fbc47bf8-qljfd\" (UID: \"e6596df5-5301-45a5-9116-dc2a4268bc80\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd" Sep 08 10:10:59.650532 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.650500 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd" Sep 08 10:10:59.932311 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:10:59.932280 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd"] Sep 08 10:10:59.933161 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:10:59.933134 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6596df5_5301_45a5_9116_dc2a4268bc80.slice/crio-4d56a283f44df953d2131a47c981f1f5dee45759956a13198bcc99f29b2a5092 WatchSource:0}: Error finding container 4d56a283f44df953d2131a47c981f1f5dee45759956a13198bcc99f29b2a5092: Status 404 returned error can't find the container with id 4d56a283f44df953d2131a47c981f1f5dee45759956a13198bcc99f29b2a5092 Sep 08 10:11:00.402226 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:00.402193 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd" event={"ID":"e6596df5-5301-45a5-9116-dc2a4268bc80","Type":"ContainerStarted","Data":"4d56a283f44df953d2131a47c981f1f5dee45759956a13198bcc99f29b2a5092"} Sep 08 10:11:04.434138 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:04.433767 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" event={"ID":"743cdc55-c328-4650-b2f5-f0bdff2c89ee","Type":"ContainerStarted","Data":"eff8c101ae0a4304b0bd2c7e3d9e7fd8d6c380b211ebe5c87008a4baa68234c4"} Sep 08 10:11:04.434138 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:04.433822 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:11:04.462415 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:04.461893 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" podStartSLOduration=1.334814686 podStartE2EDuration="6.461873557s" totalImagesPullingTime="5.127058871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:11:04.460199778 +0000 UTC m=+574.249251582" watchObservedRunningTime="2026-09-08 10:11:04.461873557 +0000 UTC m=+574.250925361" Sep 08 10:11:06.446935 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:06.446896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd" event={"ID":"e6596df5-5301-45a5-9116-dc2a4268bc80","Type":"ContainerStarted","Data":"092f00aafb00682731a96035500ecfa4183e9e659da2b59d23dd0a956f9bdfc1"} Sep 08 10:11:06.447416 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:06.446990 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd" Sep 08 10:11:08.989573 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:08.989461 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2m56t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:42820->10.134.0.44:15021: read: connection reset by peer" start-of-body= Sep 08 10:11:08.989573 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:08.989523 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" podUID="48567896-0f21-404c-863a-1ece4631e205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:42820->10.134.0.44:15021: read: connection reset by peer" Sep 08 10:11:09.465343 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:09.465311 2576 generic.go:356] "Generic (PLEG): container finished" podID="48567896-0f21-404c-863a-1ece4631e205" containerID="b05ad3d4161e7207f970598ae7abd1207c1948fed24f0eca4e0a66dc48f8a6d6" exitCode=0 Sep 08 10:11:09.465343 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:09.465351 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" event={"ID":"48567896-0f21-404c-863a-1ece4631e205","Type":"ContainerDied","Data":"b05ad3d4161e7207f970598ae7abd1207c1948fed24f0eca4e0a66dc48f8a6d6"} Sep 08 10:11:09.465802 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:09.465783 2576 scope.go:117] "RemoveContainer" containerID="b05ad3d4161e7207f970598ae7abd1207c1948fed24f0eca4e0a66dc48f8a6d6" Sep 08 10:11:09.466797 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:09.466764 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:11:09.466879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:09.466835 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:11:09.466879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:09.466866 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:11:09.502277 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:09.502199 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd" podStartSLOduration=4.874883599 podStartE2EDuration="10.502179429s" totalImagesPullingTime="5.62729583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:11:06.479292936 +0000 UTC m=+576.268344742" watchObservedRunningTime="2026-09-08 10:11:09.502179429 +0000 UTC m=+579.291231234" Sep 08 10:11:10.445660 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:10.445613 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:11:10.472942 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:10.472911 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" event={"ID":"48567896-0f21-404c-863a-1ece4631e205","Type":"ContainerStarted","Data":"94aacb65a34a98f6e93b7b044b0d26900f9ec3824b096ea2ea2cb2d6151ae35d"} Sep 08 10:11:10.759190 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:10.759104 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:11:10.759365 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:10.759340 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2m56t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 08 10:11:10.759440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:10.759398 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" podUID="48567896-0f21-404c-863a-1ece4631e205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 08 10:11:11.729044 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:11.728997 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:11:11.759149 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:11.759114 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2m56t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 08 10:11:11.759338 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:11.759177 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" podUID="48567896-0f21-404c-863a-1ece4631e205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 08 10:11:12.151038 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:12.151004 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:11:12.759884 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:12.759842 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2m56t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 08 10:11:12.760295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:12.759936 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" podUID="48567896-0f21-404c-863a-1ece4631e205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 08 10:11:13.759161 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:13.759124 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2m56t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 08 10:11:13.759332 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:13.759178 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" podUID="48567896-0f21-404c-863a-1ece4631e205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 08 10:11:14.759832 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:14.759793 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2m56t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 08 10:11:14.760223 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:14.759853 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" podUID="48567896-0f21-404c-863a-1ece4631e205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 08 10:11:15.762208 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:15.762177 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:11:15.762648 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:15.762433 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:11:15.763008 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:15.762994 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2m56t" Sep 08 10:11:17.454651 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:17.454604 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qljfd" Sep 08 10:11:26.950885 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:26.950845 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr"] Sep 08 10:11:26.954040 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:26.954023 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:26.957246 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:26.957223 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 08 10:11:26.957381 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:26.957248 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fhwp9\"" Sep 08 10:11:26.957381 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:26.957282 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 08 10:11:26.957491 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:26.957390 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 08 10:11:26.967018 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:26.966996 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr"] Sep 08 10:11:27.044493 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.044442 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:11:27.094791 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.094755 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f3579d-8372-46cf-8cb3-466410fe4552-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.094970 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.094905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6t97\" (UniqueName: \"kubernetes.io/projected/46f3579d-8372-46cf-8cb3-466410fe4552-kube-api-access-n6t97\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.095025 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.094998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f3579d-8372-46cf-8cb3-466410fe4552-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.095025 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.095022 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f3579d-8372-46cf-8cb3-466410fe4552-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.095136 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.095049 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f3579d-8372-46cf-8cb3-466410fe4552-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.095136 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.095066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f3579d-8372-46cf-8cb3-466410fe4552-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.196285 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.196252 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f3579d-8372-46cf-8cb3-466410fe4552-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.196484 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.196297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f3579d-8372-46cf-8cb3-466410fe4552-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.196484 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.196363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f3579d-8372-46cf-8cb3-466410fe4552-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.196484 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.196394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f3579d-8372-46cf-8cb3-466410fe4552-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.196484 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.196419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f3579d-8372-46cf-8cb3-466410fe4552-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.196802 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.196496 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6t97\" (UniqueName: \"kubernetes.io/projected/46f3579d-8372-46cf-8cb3-466410fe4552-kube-api-access-n6t97\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.196802 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.196739 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f3579d-8372-46cf-8cb3-466410fe4552-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.196802 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.196780 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f3579d-8372-46cf-8cb3-466410fe4552-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.196977 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.196835 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f3579d-8372-46cf-8cb3-466410fe4552-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.198759 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.198741 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f3579d-8372-46cf-8cb3-466410fe4552-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.199054 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.199033 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f3579d-8372-46cf-8cb3-466410fe4552-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.204719 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.204668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6t97\" (UniqueName: \"kubernetes.io/projected/46f3579d-8372-46cf-8cb3-466410fe4552-kube-api-access-n6t97\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr\" (UID: \"46f3579d-8372-46cf-8cb3-466410fe4552\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.264300 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.264262 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:27.529456 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.529385 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:11:27.567593 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:27.567549 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr"] Sep 08 10:11:27.574611 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:11:27.574577 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f3579d_8372_46cf_8cb3_466410fe4552.slice/crio-d45780434f4cb3052bd47d5e24e7fdaf1d4936762f0faadc29de781fe8eca288 WatchSource:0}: Error finding container d45780434f4cb3052bd47d5e24e7fdaf1d4936762f0faadc29de781fe8eca288: Status 404 returned error can't find the container with id d45780434f4cb3052bd47d5e24e7fdaf1d4936762f0faadc29de781fe8eca288 Sep 08 10:11:28.580870 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:28.580824 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" event={"ID":"46f3579d-8372-46cf-8cb3-466410fe4552","Type":"ContainerStarted","Data":"d45780434f4cb3052bd47d5e24e7fdaf1d4936762f0faadc29de781fe8eca288"} Sep 08 10:11:29.587247 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:29.587214 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" event={"ID":"46f3579d-8372-46cf-8cb3-466410fe4552","Type":"ContainerStarted","Data":"473d62aff0b641627f3a5abc70004f41051af446cd76a1febe925cc14612984e"} Sep 08 10:11:29.616857 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:29.616778 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" podStartSLOduration=2.5522617309999998 podStartE2EDuration="3.616763252s" totalImagesPullingTime="1.064501521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:11:29.613573403 +0000 UTC m=+599.402625206" watchObservedRunningTime="2026-09-08 10:11:29.616763252 +0000 UTC m=+599.405815120" Sep 08 10:11:36.469955 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:36.469922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:11:36.574909 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:36.574875 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:11:37.264534 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:37.264502 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:37.264729 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:37.264564 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:37.276880 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:37.276849 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:37.643233 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:37.643205 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr" Sep 08 10:11:38.265476 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:38.265440 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:11:43.385469 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:43.385441 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:11:44.381533 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:44.381500 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:11:49.503744 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:49.503704 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:11:50.845049 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:50.845014 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:11:53.351778 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:53.351694 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:11:55.557640 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:55.557585 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:11:59.610713 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:11:59.610668 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:12:02.355656 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:12:02.355611 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:12:07.126247 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:12:07.126208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:12:09.881232 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:12:09.881142 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:12:25.148091 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:12:25.148056 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:00.840813 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:00.840785 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:04.167345 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:04.167304 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:13.330378 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:13.330347 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:27.557215 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:27.556992 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:27.652006 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:27.651960 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:28.771219 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:28.771175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:28.864522 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:28.864492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:46.901636 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:46.901581 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:47.041808 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.041769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:47.287760 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.287678 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc"] Sep 08 10:13:47.291730 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.291641 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.295022 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.295000 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c28013-openshift-default-dockercfg-7hrfj\"" Sep 08 10:13:47.318501 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.318474 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc"] Sep 08 10:13:47.340993 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.340964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-podinfo\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.341238 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.341042 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-token\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.341238 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.341121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpnr2\" (UniqueName: \"kubernetes.io/projected/cc97ae01-9034-497d-9b1b-e34c8997e485-kube-api-access-jpnr2\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.341238 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.341161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-workload-socket\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.341238 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.341216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-credential-socket\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.341483 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.341296 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-envoy\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.341483 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.341322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-data\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.341483 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.341353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc97ae01-9034-497d-9b1b-e34c8997e485-istiod-ca-cert\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.341483 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.341389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-workload-certs\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.343199 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.343175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:47.442416 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.442375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-envoy\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.442577 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.442424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-data\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.442577 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.442453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc97ae01-9034-497d-9b1b-e34c8997e485-istiod-ca-cert\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.442577 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.442489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-workload-certs\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.442577 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.442545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-podinfo\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.442837 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.442578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-token\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.442837 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.442655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpnr2\" (UniqueName: \"kubernetes.io/projected/cc97ae01-9034-497d-9b1b-e34c8997e485-kube-api-access-jpnr2\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.442837 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.442692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-workload-socket\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.442998 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.442935 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-data\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.443058 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.443040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-credential-socket\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.443472 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.443274 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc97ae01-9034-497d-9b1b-e34c8997e485-istiod-ca-cert\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.443472 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.443388 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-workload-socket\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.443472 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.443408 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-workload-certs\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.443472 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.443461 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-credential-socket\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.445113 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.445094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-podinfo\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.445234 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.445216 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-envoy\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.451307 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.451279 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-token\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.451307 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.451300 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpnr2\" (UniqueName: \"kubernetes.io/projected/cc97ae01-9034-497d-9b1b-e34c8997e485-kube-api-access-jpnr2\") pod \"e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.605121 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.605040 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:47.800848 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.800810 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:47.888230 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.888207 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc"] Sep 08 10:13:47.890660 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:13:47.890270 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc97ae01_9034_497d_9b1b_e34c8997e485.slice/crio-9c4e3f8c4d5df6b3099886af419fc72710a21f6f6b40dbc1d9ecc88e7d6d98d3 WatchSource:0}: Error finding container 9c4e3f8c4d5df6b3099886af419fc72710a21f6f6b40dbc1d9ecc88e7d6d98d3: Status 404 returned error can't find the container with id 9c4e3f8c4d5df6b3099886af419fc72710a21f6f6b40dbc1d9ecc88e7d6d98d3 Sep 08 10:13:47.892604 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.892566 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:13:47.892736 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.892673 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:13:47.892736 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:47.892725 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:13:48.360362 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:48.360288 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:48.366071 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:48.366036 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" event={"ID":"cc97ae01-9034-497d-9b1b-e34c8997e485","Type":"ContainerStarted","Data":"e3a9dfb7938225fa18aa151de3c195b1b4c31e3deafbef23141df43685f5c103"} Sep 08 10:13:48.366215 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:48.366097 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" event={"ID":"cc97ae01-9034-497d-9b1b-e34c8997e485","Type":"ContainerStarted","Data":"9c4e3f8c4d5df6b3099886af419fc72710a21f6f6b40dbc1d9ecc88e7d6d98d3"} Sep 08 10:13:48.417790 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:48.417735 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" podStartSLOduration=1.417718725 podStartE2EDuration="1.417718725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:13:48.414475207 +0000 UTC m=+738.203527019" watchObservedRunningTime="2026-09-08 10:13:48.417718725 +0000 UTC m=+738.206770529" Sep 08 10:13:48.491141 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:48.491107 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:48.606269 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:48.606232 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:48.610902 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:48.610846 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:48.849190 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:48.849158 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7899-140710403925168"] Sep 08 10:13:48.852679 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:48.852656 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-140710403925168" Sep 08 10:13:48.854807 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:48.854789 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5nsxf\"" Sep 08 10:13:48.858774 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:48.858750 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-140710403925168"] Sep 08 10:13:48.958030 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:48.958003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlnhr\" (UniqueName: \"kubernetes.io/projected/15a79b8e-e929-475c-9734-aba218eaa8f3-kube-api-access-xlnhr\") pod \"test-curl-7899-140710403925168\" (UID: \"15a79b8e-e929-475c-9734-aba218eaa8f3\") " pod="opendatahub/test-curl-7899-140710403925168" Sep 08 10:13:49.059059 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:49.059034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlnhr\" (UniqueName: \"kubernetes.io/projected/15a79b8e-e929-475c-9734-aba218eaa8f3-kube-api-access-xlnhr\") pod \"test-curl-7899-140710403925168\" (UID: \"15a79b8e-e929-475c-9734-aba218eaa8f3\") " pod="opendatahub/test-curl-7899-140710403925168" Sep 08 10:13:49.067866 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:49.067840 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlnhr\" (UniqueName: \"kubernetes.io/projected/15a79b8e-e929-475c-9734-aba218eaa8f3-kube-api-access-xlnhr\") pod \"test-curl-7899-140710403925168\" (UID: \"15a79b8e-e929-475c-9734-aba218eaa8f3\") " pod="opendatahub/test-curl-7899-140710403925168" Sep 08 10:13:49.164692 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:49.164653 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-140710403925168" Sep 08 10:13:49.372085 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:49.372054 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:49.373061 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:49.373036 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:13:49.438311 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:49.438285 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-140710403925168"] Sep 08 10:13:49.439939 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:13:49.439898 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15a79b8e_e929_475c_9734_aba218eaa8f3.slice/crio-1ae299a42ac852ba256fe73517897b4e65b5bf94cc2e1d5cf405078ca6c6164b WatchSource:0}: Error finding container 1ae299a42ac852ba256fe73517897b4e65b5bf94cc2e1d5cf405078ca6c6164b: Status 404 returned error can't find the container with id 1ae299a42ac852ba256fe73517897b4e65b5bf94cc2e1d5cf405078ca6c6164b Sep 08 10:13:49.441407 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:49.441389 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 10:13:50.378390 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:50.378307 2576 generic.go:356] "Generic (PLEG): container finished" podID="15a79b8e-e929-475c-9734-aba218eaa8f3" containerID="6af6779fab9fd585be10c29b3d5bcf6515df8bd82b0fa9e0211b879b7786badb" exitCode=0 Sep 08 10:13:50.378390 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:50.378382 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-140710403925168" event={"ID":"15a79b8e-e929-475c-9734-aba218eaa8f3","Type":"ContainerDied","Data":"6af6779fab9fd585be10c29b3d5bcf6515df8bd82b0fa9e0211b879b7786badb"} Sep 08 10:13:50.378819 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:50.378418 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-140710403925168" event={"ID":"15a79b8e-e929-475c-9734-aba218eaa8f3","Type":"ContainerStarted","Data":"1ae299a42ac852ba256fe73517897b4e65b5bf94cc2e1d5cf405078ca6c6164b"} Sep 08 10:13:51.036271 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.036239 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:51.061140 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.061115 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:13:51.142328 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.142297 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:51.526180 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.526157 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-140710403925168" Sep 08 10:13:51.550009 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.549974 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7899-140710403925168_15a79b8e-e929-475c-9734-aba218eaa8f3/test-curl-7899-140710403925168/0.log" Sep 08 10:13:51.572587 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.572560 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh"] Sep 08 10:13:51.573061 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.573043 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a79b8e-e929-475c-9734-aba218eaa8f3" containerName="test-curl-7899-140710403925168" Sep 08 10:13:51.573143 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.573063 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a79b8e-e929-475c-9734-aba218eaa8f3" containerName="test-curl-7899-140710403925168" Sep 08 10:13:51.573202 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.573151 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a79b8e-e929-475c-9734-aba218eaa8f3" containerName="test-curl-7899-140710403925168" Sep 08 10:13:51.576776 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.576750 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:51.578546 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.578520 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7899-140710403925168"] Sep 08 10:13:51.579361 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.579275 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c28013-serving-cert\"" Sep 08 10:13:51.579361 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.579342 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w6wth\"" Sep 08 10:13:51.580853 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.580821 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlnhr\" (UniqueName: \"kubernetes.io/projected/15a79b8e-e929-475c-9734-aba218eaa8f3-kube-api-access-xlnhr\") pod \"15a79b8e-e929-475c-9734-aba218eaa8f3\" (UID: \"15a79b8e-e929-475c-9734-aba218eaa8f3\") " Sep 08 10:13:51.585067 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.584648 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a79b8e-e929-475c-9734-aba218eaa8f3-kube-api-access-xlnhr" (OuterVolumeSpecName: "kube-api-access-xlnhr") pod "15a79b8e-e929-475c-9734-aba218eaa8f3" (UID: "15a79b8e-e929-475c-9734-aba218eaa8f3"). InnerVolumeSpecName "kube-api-access-xlnhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:13:51.585067 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.584968 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7899-140710403925168"] Sep 08 10:13:51.588278 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.588251 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh"] Sep 08 10:13:51.608248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.608223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:51.682212 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.682178 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad79fe7-9c86-4583-88f0-86ec392540e7-maas-api-tls\") pod \"maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:51.682382 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.682225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6wz2\" (UniqueName: \"kubernetes.io/projected/6ad79fe7-9c86-4583-88f0-86ec392540e7-kube-api-access-v6wz2\") pod \"maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:51.682382 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.682258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ad79fe7-9c86-4583-88f0-86ec392540e7-metrics-certs\") pod \"maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:51.682382 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.682335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad79fe7-9c86-4583-88f0-86ec392540e7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:51.682505 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.682467 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlnhr\" (UniqueName: \"kubernetes.io/projected/15a79b8e-e929-475c-9734-aba218eaa8f3-kube-api-access-xlnhr\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:13:51.687604 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.687580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:51.693794 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.693769 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7899-140710403925168"] Sep 08 10:13:51.697805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.697786 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-140710403925168" Sep 08 10:13:51.720955 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.720920 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-140710403925168"] Sep 08 10:13:51.783415 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.783213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad79fe7-9c86-4583-88f0-86ec392540e7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:51.783415 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.783348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad79fe7-9c86-4583-88f0-86ec392540e7-maas-api-tls\") pod \"maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:51.783415 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.783379 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6wz2\" (UniqueName: \"kubernetes.io/projected/6ad79fe7-9c86-4583-88f0-86ec392540e7-kube-api-access-v6wz2\") pod \"maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:51.783415 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.783407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ad79fe7-9c86-4583-88f0-86ec392540e7-metrics-certs\") pod \"maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:51.783917 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.783463 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2wm5\" (UniqueName: \"kubernetes.io/projected/711ce81f-4e83-4393-90f1-fd5ee3a5ae7c-kube-api-access-l2wm5\") pod \"test-curl-7899-140710403925168\" (UID: \"711ce81f-4e83-4393-90f1-fd5ee3a5ae7c\") " pod="opendatahub/test-curl-7899-140710403925168" Sep 08 10:13:51.784058 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.784023 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad79fe7-9c86-4583-88f0-86ec392540e7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:51.785964 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.785940 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad79fe7-9c86-4583-88f0-86ec392540e7-maas-api-tls\") pod \"maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:51.786049 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.785989 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ad79fe7-9c86-4583-88f0-86ec392540e7-metrics-certs\") pod \"maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:51.791552 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.791525 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6wz2\" (UniqueName: \"kubernetes.io/projected/6ad79fe7-9c86-4583-88f0-86ec392540e7-kube-api-access-v6wz2\") pod \"maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:51.832124 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.832096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:51.884544 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.884509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2wm5\" (UniqueName: \"kubernetes.io/projected/711ce81f-4e83-4393-90f1-fd5ee3a5ae7c-kube-api-access-l2wm5\") pod \"test-curl-7899-140710403925168\" (UID: \"711ce81f-4e83-4393-90f1-fd5ee3a5ae7c\") " pod="opendatahub/test-curl-7899-140710403925168" Sep 08 10:13:51.893039 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.893003 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2wm5\" (UniqueName: \"kubernetes.io/projected/711ce81f-4e83-4393-90f1-fd5ee3a5ae7c-kube-api-access-l2wm5\") pod \"test-curl-7899-140710403925168\" (UID: \"711ce81f-4e83-4393-90f1-fd5ee3a5ae7c\") " pod="opendatahub/test-curl-7899-140710403925168" Sep 08 10:13:51.902371 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:51.902345 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:52.021366 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:52.021338 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-140710403925168" Sep 08 10:13:52.183446 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:52.183422 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh"] Sep 08 10:13:52.184689 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:13:52.184661 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad79fe7_9c86_4583_88f0_86ec392540e7.slice/crio-3441778e912d9868d88a4bc1b7000e3e23dad87969b501f7afaa7ac177588a9b WatchSource:0}: Error finding container 3441778e912d9868d88a4bc1b7000e3e23dad87969b501f7afaa7ac177588a9b: Status 404 returned error can't find the container with id 3441778e912d9868d88a4bc1b7000e3e23dad87969b501f7afaa7ac177588a9b Sep 08 10:13:52.224493 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:52.224405 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:52.297096 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:52.297064 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-140710403925168"] Sep 08 10:13:52.298097 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:13:52.298073 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod711ce81f_4e83_4393_90f1_fd5ee3a5ae7c.slice/crio-3778b5de58561148e7a3a22fde76176636cb9328536a8763a68d66d9a85054b4 WatchSource:0}: Error finding container 3778b5de58561148e7a3a22fde76176636cb9328536a8763a68d66d9a85054b4: Status 404 returned error can't find the container with id 3778b5de58561148e7a3a22fde76176636cb9328536a8763a68d66d9a85054b4 Sep 08 10:13:52.299330 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:52.299306 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:52.391598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:52.391568 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ae299a42ac852ba256fe73517897b4e65b5bf94cc2e1d5cf405078ca6c6164b" Sep 08 10:13:52.391598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:52.391569 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-140710403925168" Sep 08 10:13:52.393019 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:52.392989 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-140710403925168" event={"ID":"711ce81f-4e83-4393-90f1-fd5ee3a5ae7c","Type":"ContainerStarted","Data":"3778b5de58561148e7a3a22fde76176636cb9328536a8763a68d66d9a85054b4"} Sep 08 10:13:52.394359 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:52.394328 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" event={"ID":"6ad79fe7-9c86-4583-88f0-86ec392540e7","Type":"ContainerStarted","Data":"3441778e912d9868d88a4bc1b7000e3e23dad87969b501f7afaa7ac177588a9b"} Sep 08 10:13:52.403186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:52.403155 2576 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7899-140710403925168" oldPodUID="15a79b8e-e929-475c-9734-aba218eaa8f3" podUID="711ce81f-4e83-4393-90f1-fd5ee3a5ae7c" Sep 08 10:13:52.680156 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:52.680113 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15a79b8e-e929-475c-9734-aba218eaa8f3" path="/var/lib/kubelet/pods/15a79b8e-e929-475c-9734-aba218eaa8f3/volumes" Sep 08 10:13:53.402557 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:53.402523 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" event={"ID":"6ad79fe7-9c86-4583-88f0-86ec392540e7","Type":"ContainerStarted","Data":"83b2d16ab06f7b086a0234731dee91297999e56c2856415316940789ea0cab96"} Sep 08 10:13:53.402767 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:53.402609 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:53.422395 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:53.422344 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" podStartSLOduration=1.490372111 podStartE2EDuration="2.422329731s" totalImagesPullingTime="931.95762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:13:53.418898067 +0000 UTC m=+743.207949865" watchObservedRunningTime="2026-09-08 10:13:53.422329731 +0000 UTC m=+743.211381535" Sep 08 10:13:54.411007 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:54.410973 2576 generic.go:356] "Generic (PLEG): container finished" podID="711ce81f-4e83-4393-90f1-fd5ee3a5ae7c" containerID="f97888d5e844efff430d289e471d558f10b09de1eb50622946f84c4a30cc3c32" exitCode=0 Sep 08 10:13:54.411428 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:54.411067 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-140710403925168" event={"ID":"711ce81f-4e83-4393-90f1-fd5ee3a5ae7c","Type":"ContainerDied","Data":"f97888d5e844efff430d289e471d558f10b09de1eb50622946f84c4a30cc3c32"} Sep 08 10:13:55.104856 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:55.104817 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:55.170269 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:55.170229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:55.317653 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:55.317606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:55.556534 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:55.556510 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-140710403925168" Sep 08 10:13:55.597003 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:55.596975 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7899-140710403925168_711ce81f-4e83-4393-90f1-fd5ee3a5ae7c/test-curl-7899-140710403925168/0.log" Sep 08 10:13:55.630333 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:55.630298 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7899-140710403925168"] Sep 08 10:13:55.635673 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:55.635648 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7899-140710403925168"] Sep 08 10:13:55.721664 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:55.721561 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2wm5\" (UniqueName: \"kubernetes.io/projected/711ce81f-4e83-4393-90f1-fd5ee3a5ae7c-kube-api-access-l2wm5\") pod \"711ce81f-4e83-4393-90f1-fd5ee3a5ae7c\" (UID: \"711ce81f-4e83-4393-90f1-fd5ee3a5ae7c\") " Sep 08 10:13:55.723740 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:55.723706 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711ce81f-4e83-4393-90f1-fd5ee3a5ae7c-kube-api-access-l2wm5" (OuterVolumeSpecName: "kube-api-access-l2wm5") pod "711ce81f-4e83-4393-90f1-fd5ee3a5ae7c" (UID: "711ce81f-4e83-4393-90f1-fd5ee3a5ae7c"). InnerVolumeSpecName "kube-api-access-l2wm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:13:55.822991 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:55.822958 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2wm5\" (UniqueName: \"kubernetes.io/projected/711ce81f-4e83-4393-90f1-fd5ee3a5ae7c-kube-api-access-l2wm5\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:13:56.391568 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.391294 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr"] Sep 08 10:13:56.392019 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.391986 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="711ce81f-4e83-4393-90f1-fd5ee3a5ae7c" containerName="test-curl-7899-140710403925168" Sep 08 10:13:56.392019 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.392010 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="711ce81f-4e83-4393-90f1-fd5ee3a5ae7c" containerName="test-curl-7899-140710403925168" Sep 08 10:13:56.392259 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.392126 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="711ce81f-4e83-4393-90f1-fd5ee3a5ae7c" containerName="test-curl-7899-140710403925168" Sep 08 10:13:56.395283 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.395259 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.397718 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.397695 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-338aba-openshift-default-dockercfg-t2d8x\"" Sep 08 10:13:56.413503 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.413477 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr"] Sep 08 10:13:56.426591 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.426573 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-140710403925168" Sep 08 10:13:56.426723 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.426578 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3778b5de58561148e7a3a22fde76176636cb9328536a8763a68d66d9a85054b4" Sep 08 10:13:56.437630 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:13:56.437589 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7899-140710403925168\" is forbidden: User \"system:node:ip-10-0-132-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-206.ec2.internal' and this object" podUID="711ce81f-4e83-4393-90f1-fd5ee3a5ae7c" pod="opendatahub/test-curl-7899-140710403925168" Sep 08 10:13:56.530498 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.530462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-workload-certs\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.530737 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.530572 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ba59c2-f394-4365-84e5-056739f115cc-istio-token\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.530737 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.530605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ba59c2-f394-4365-84e5-056739f115cc-istio-podinfo\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.530737 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.530660 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzzzn\" (UniqueName: \"kubernetes.io/projected/d9ba59c2-f394-4365-84e5-056739f115cc-kube-api-access-kzzzn\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.530889 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.530742 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-istio-data\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.530889 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.530772 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ba59c2-f394-4365-84e5-056739f115cc-istiod-ca-cert\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.530889 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.530848 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-credential-socket\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.530998 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.530909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-workload-socket\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.530998 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.530937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-istio-envoy\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.631671 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.631612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-istio-envoy\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.632063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.631741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-workload-certs\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.632063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.631804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ba59c2-f394-4365-84e5-056739f115cc-istio-token\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.632063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.631834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ba59c2-f394-4365-84e5-056739f115cc-istio-podinfo\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.632063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.631869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzzzn\" (UniqueName: \"kubernetes.io/projected/d9ba59c2-f394-4365-84e5-056739f115cc-kube-api-access-kzzzn\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.632063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.631900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-istio-data\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.632063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.631922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ba59c2-f394-4365-84e5-056739f115cc-istiod-ca-cert\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.632063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.631970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-credential-socket\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.632063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.632023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-workload-socket\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.632455 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.632182 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-workload-certs\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.632455 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.632302 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-istio-data\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.632455 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.632389 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-workload-socket\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.632455 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.632443 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-credential-socket\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.632730 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.632713 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ba59c2-f394-4365-84e5-056739f115cc-istiod-ca-cert\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.634192 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.634160 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-istio-envoy\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.634516 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.634494 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ba59c2-f394-4365-84e5-056739f115cc-istio-podinfo\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.640430 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.640408 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ba59c2-f394-4365-84e5-056739f115cc-istio-token\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.640574 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.640550 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzzzn\" (UniqueName: \"kubernetes.io/projected/d9ba59c2-f394-4365-84e5-056739f115cc-kube-api-access-kzzzn\") pod \"e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.681670 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.677863 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="711ce81f-4e83-4393-90f1-fd5ee3a5ae7c" path="/var/lib/kubelet/pods/711ce81f-4e83-4393-90f1-fd5ee3a5ae7c/volumes" Sep 08 10:13:56.706345 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.706315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:56.982258 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.982236 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr"] Sep 08 10:13:56.983539 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:13:56.983512 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ba59c2_f394_4365_84e5_056739f115cc.slice/crio-033e55bf235df91b41a3465a9f91f5aeafe427760a04277dea8d48d43bf052fb WatchSource:0}: Error finding container 033e55bf235df91b41a3465a9f91f5aeafe427760a04277dea8d48d43bf052fb: Status 404 returned error can't find the container with id 033e55bf235df91b41a3465a9f91f5aeafe427760a04277dea8d48d43bf052fb Sep 08 10:13:56.985901 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.985869 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:13:56.986014 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.985949 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:13:56.986014 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:56.985995 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:13:57.437096 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:57.437063 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" event={"ID":"d9ba59c2-f394-4365-84e5-056739f115cc","Type":"ContainerStarted","Data":"3c6d3e26d3a28b4d29ad71c8865de9496ac6505760923ae4348ded34482447ed"} Sep 08 10:13:57.437247 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:57.437114 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" event={"ID":"d9ba59c2-f394-4365-84e5-056739f115cc","Type":"ContainerStarted","Data":"033e55bf235df91b41a3465a9f91f5aeafe427760a04277dea8d48d43bf052fb"} Sep 08 10:13:57.459853 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:57.459807 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" podStartSLOduration=1.459792311 podStartE2EDuration="1.459792311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:13:57.456730405 +0000 UTC m=+747.245782210" watchObservedRunningTime="2026-09-08 10:13:57.459792311 +0000 UTC m=+747.248844115" Sep 08 10:13:57.707189 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:57.707104 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:57.835285 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:57.835258 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:57.935764 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:57.935733 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:58.274317 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:58.274274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:58.364919 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:58.364891 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:58.712377 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:58.712347 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:59.420396 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:59.420366 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:13:59.449891 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:59.449854 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:59.450837 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:59.450816 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:13:59.873214 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:59.873127 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s"] Sep 08 10:13:59.877114 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:59.877092 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:13:59.878131 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:59.878110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:13:59.881553 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:59.881536 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-cbf4760c-serving-cert\"" Sep 08 10:13:59.964159 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:59.964121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp6nw\" (UniqueName: \"kubernetes.io/projected/bad9eb9a-97b9-4599-9c26-aed809766f1d-kube-api-access-hp6nw\") pod \"maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:13:59.964436 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:59.964411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bad9eb9a-97b9-4599-9c26-aed809766f1d-openshift-service-ca\") pod \"maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:13:59.964587 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:59.964570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bad9eb9a-97b9-4599-9c26-aed809766f1d-maas-api-tls\") pod \"maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:13:59.964806 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:13:59.964787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad9eb9a-97b9-4599-9c26-aed809766f1d-metrics-certs\") pod \"maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:14:00.023882 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:00.023851 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s"] Sep 08 10:14:00.065821 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:00.065790 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp6nw\" (UniqueName: \"kubernetes.io/projected/bad9eb9a-97b9-4599-9c26-aed809766f1d-kube-api-access-hp6nw\") pod \"maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:14:00.066020 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:00.065836 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bad9eb9a-97b9-4599-9c26-aed809766f1d-openshift-service-ca\") pod \"maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:14:00.066020 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:00.065866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bad9eb9a-97b9-4599-9c26-aed809766f1d-maas-api-tls\") pod \"maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:14:00.066152 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:00.066063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad9eb9a-97b9-4599-9c26-aed809766f1d-metrics-certs\") pod \"maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:14:00.066214 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:14:00.066190 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-serving-cert: secret "maas-api-e2e-ait-cbf4760c-serving-cert" not found Sep 08 10:14:00.066292 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:14:00.066277 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad9eb9a-97b9-4599-9c26-aed809766f1d-maas-api-tls podName:bad9eb9a-97b9-4599-9c26-aed809766f1d nodeName:}" failed. No retries permitted until 2026-09-08 10:14:00.566255754 +0000 UTC m=+750.355307548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bad9eb9a-97b9-4599-9c26-aed809766f1d-maas-api-tls") pod "maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" (UID: "bad9eb9a-97b9-4599-9c26-aed809766f1d") : secret "maas-api-e2e-ait-cbf4760c-serving-cert" not found Sep 08 10:14:00.067074 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:00.067050 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bad9eb9a-97b9-4599-9c26-aed809766f1d-openshift-service-ca\") pod \"maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:14:00.068902 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:00.068885 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad9eb9a-97b9-4599-9c26-aed809766f1d-metrics-certs\") pod \"maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:14:00.116582 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:00.116550 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp6nw\" (UniqueName: \"kubernetes.io/projected/bad9eb9a-97b9-4599-9c26-aed809766f1d-kube-api-access-hp6nw\") pod \"maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:14:00.124658 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:00.124552 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:00.399112 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:00.399069 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:00.571540 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:00.571498 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bad9eb9a-97b9-4599-9c26-aed809766f1d-maas-api-tls\") pod \"maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:14:00.574092 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:00.574067 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bad9eb9a-97b9-4599-9c26-aed809766f1d-maas-api-tls\") pod \"maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:14:00.729387 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:00.729280 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:00.787942 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:00.787915 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:14:01.103699 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:01.102898 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s"] Sep 08 10:14:01.174092 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:01.174039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:01.272206 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:01.272168 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:01.462968 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:01.462930 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" event={"ID":"bad9eb9a-97b9-4599-9c26-aed809766f1d","Type":"ContainerStarted","Data":"8e61687c869d8f258bd7b97101e8081f01e3260224e8cff5f73ea60e1595fc36"} Sep 08 10:14:01.620563 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:01.620538 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:02.285003 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:02.284972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:02.388868 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:02.388827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:03.503662 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:03.503629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:04.481827 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:04.481793 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" event={"ID":"bad9eb9a-97b9-4599-9c26-aed809766f1d","Type":"ContainerStarted","Data":"25dc34dca0fe525b484a9ee8f761dc760d5b6defcbc3f33b280792b2a4e4bc89"} Sep 08 10:14:04.482016 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:04.481927 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:14:04.506550 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:04.506480 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" podStartSLOduration=2.71796494 podStartE2EDuration="5.506456023s" totalImagesPullingTime="2.788491083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:13:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:14:04.501580452 +0000 UTC m=+754.290632305" watchObservedRunningTime="2026-09-08 10:14:04.506456023 +0000 UTC m=+754.295507832" Sep 08 10:14:06.358844 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:06.358809 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:14:07.751791 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:07.751752 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:07.868359 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:07.868333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:07.957969 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:07.957840 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:08.056964 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:08.056883 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:08.513739 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:08.512305 2576 generic.go:356] "Generic (PLEG): container finished" podID="cc97ae01-9034-497d-9b1b-e34c8997e485" containerID="e3a9dfb7938225fa18aa151de3c195b1b4c31e3deafbef23141df43685f5c103" exitCode=0 Sep 08 10:14:08.513739 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:08.512502 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" event={"ID":"cc97ae01-9034-497d-9b1b-e34c8997e485","Type":"ContainerDied","Data":"e3a9dfb7938225fa18aa151de3c195b1b4c31e3deafbef23141df43685f5c103"} Sep 08 10:14:08.513739 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:08.513075 2576 scope.go:117] "RemoveContainer" containerID="e3a9dfb7938225fa18aa151de3c195b1b4c31e3deafbef23141df43685f5c103" Sep 08 10:14:08.515930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:08.515524 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:14:08.515930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:08.515639 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:14:08.515930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:08.515689 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:14:09.522687 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:09.522650 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" event={"ID":"cc97ae01-9034-497d-9b1b-e34c8997e485","Type":"ContainerStarted","Data":"b5f0fe2010924b8a26c2433177760bb8003ec5927e73469ce9d3fa522960a5db"} Sep 08 10:14:09.616639 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:09.616064 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:14:09.616639 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:09.616197 2576 patch_prober.go:28] interesting pod/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 08 10:14:09.616639 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:09.616242 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" podUID="cc97ae01-9034-497d-9b1b-e34c8997e485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 08 10:14:10.508779 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:10.508744 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:14:10.614043 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:10.614005 2576 patch_prober.go:28] interesting pod/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 08 10:14:10.614598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:10.614561 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" podUID="cc97ae01-9034-497d-9b1b-e34c8997e485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 08 10:14:10.793334 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:10.793192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:11.606239 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:11.606192 2576 patch_prober.go:28] interesting pod/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 08 10:14:11.606436 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:11.606263 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" podUID="cc97ae01-9034-497d-9b1b-e34c8997e485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 08 10:14:12.486816 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:12.486787 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:12.617763 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:12.617723 2576 patch_prober.go:28] interesting pod/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 08 10:14:12.618038 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:12.618008 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" podUID="cc97ae01-9034-497d-9b1b-e34c8997e485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 08 10:14:13.606013 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:13.605970 2576 patch_prober.go:28] interesting pod/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 08 10:14:13.606475 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:13.606048 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" podUID="cc97ae01-9034-497d-9b1b-e34c8997e485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 08 10:14:13.787385 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:13.787351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:14.224498 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:14.224454 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:14.312222 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:14.312133 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:14.422164 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:14.422127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:14.605962 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:14.605920 2576 patch_prober.go:28] interesting pod/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 08 10:14:14.606160 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:14.605995 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" podUID="cc97ae01-9034-497d-9b1b-e34c8997e485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 08 10:14:15.322949 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:15.322918 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:15.492127 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:15.491776 2576 patch_prober.go:28] interesting pod/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 10:14:15.492127 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:15.491845 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" podUID="d9ba59c2-f394-4365-84e5-056739f115cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 10:14:15.609811 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:15.609726 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:14:15.610262 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:15.609982 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:14:15.610769 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:15.610750 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:14:15.641234 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:15.641202 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:16.585116 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:16.585085 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:16.687458 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:16.687420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:17.583357 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:17.583324 2576 generic.go:356] "Generic (PLEG): container finished" podID="d9ba59c2-f394-4365-84e5-056739f115cc" containerID="3c6d3e26d3a28b4d29ad71c8865de9496ac6505760923ae4348ded34482447ed" exitCode=0 Sep 08 10:14:17.583561 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:17.583397 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" event={"ID":"d9ba59c2-f394-4365-84e5-056739f115cc","Type":"ContainerDied","Data":"3c6d3e26d3a28b4d29ad71c8865de9496ac6505760923ae4348ded34482447ed"} Sep 08 10:14:17.583934 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:17.583920 2576 scope.go:117] "RemoveContainer" containerID="3c6d3e26d3a28b4d29ad71c8865de9496ac6505760923ae4348ded34482447ed" Sep 08 10:14:17.585032 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:17.585002 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:14:17.585109 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:17.585091 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:14:17.585160 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:17.585145 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:14:18.595656 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:18.595260 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" event={"ID":"d9ba59c2-f394-4365-84e5-056739f115cc","Type":"ContainerStarted","Data":"fdd36be16f4586f1780e2d2f7c18f31727b636881a6cc7702b568b8a15460861"} Sep 08 10:14:18.706464 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:18.706419 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:14:18.706948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:18.706917 2576 patch_prober.go:28] interesting pod/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 08 10:14:18.707043 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:18.706987 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" podUID="d9ba59c2-f394-4365-84e5-056739f115cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 08 10:14:19.707265 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:19.707226 2576 patch_prober.go:28] interesting pod/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 08 10:14:19.707858 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:19.707828 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" podUID="d9ba59c2-f394-4365-84e5-056739f115cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 08 10:14:20.377043 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.376924 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr"] Sep 08 10:14:20.379837 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.379815 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:20.384682 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.384663 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9e365b-serving-cert\"" Sep 08 10:14:20.431730 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.431698 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr"] Sep 08 10:14:20.488388 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.488353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0526bd46-f2e0-4c3c-8386-2043f72eab6d-maas-api-tls\") pod \"maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:20.488691 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.488669 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdvmc\" (UniqueName: \"kubernetes.io/projected/0526bd46-f2e0-4c3c-8386-2043f72eab6d-kube-api-access-mdvmc\") pod \"maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:20.488965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.488946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0526bd46-f2e0-4c3c-8386-2043f72eab6d-metrics-certs\") pod \"maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:20.489138 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.489119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0526bd46-f2e0-4c3c-8386-2043f72eab6d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:20.590152 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.590117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0526bd46-f2e0-4c3c-8386-2043f72eab6d-metrics-certs\") pod \"maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:20.590415 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.590392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0526bd46-f2e0-4c3c-8386-2043f72eab6d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:20.590585 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.590569 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0526bd46-f2e0-4c3c-8386-2043f72eab6d-maas-api-tls\") pod \"maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:20.590775 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.590757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdvmc\" (UniqueName: \"kubernetes.io/projected/0526bd46-f2e0-4c3c-8386-2043f72eab6d-kube-api-access-mdvmc\") pod \"maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:20.591772 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:14:20.591750 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-serving-cert: secret "maas-api-e2e-inf-b-9e365b-serving-cert" not found Sep 08 10:14:20.592013 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:14:20.592001 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0526bd46-f2e0-4c3c-8386-2043f72eab6d-maas-api-tls podName:0526bd46-f2e0-4c3c-8386-2043f72eab6d nodeName:}" failed. No retries permitted until 2026-09-08 10:14:21.091981018 +0000 UTC m=+770.881032808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0526bd46-f2e0-4c3c-8386-2043f72eab6d-maas-api-tls") pod "maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" (UID: "0526bd46-f2e0-4c3c-8386-2043f72eab6d") : secret "maas-api-e2e-inf-b-9e365b-serving-cert" not found Sep 08 10:14:20.601641 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.592292 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0526bd46-f2e0-4c3c-8386-2043f72eab6d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:20.602832 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.602809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0526bd46-f2e0-4c3c-8386-2043f72eab6d-metrics-certs\") pod \"maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:20.605030 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.605003 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdvmc\" (UniqueName: \"kubernetes.io/projected/0526bd46-f2e0-4c3c-8386-2043f72eab6d-kube-api-access-mdvmc\") pod \"maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:20.706975 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.706940 2576 patch_prober.go:28] interesting pod/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 08 10:14:20.707242 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:20.707211 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" podUID="d9ba59c2-f394-4365-84e5-056739f115cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 08 10:14:21.115697 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:21.112120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:21.125684 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:21.125644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0526bd46-f2e0-4c3c-8386-2043f72eab6d-maas-api-tls\") pod \"maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:21.128224 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:21.128202 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0526bd46-f2e0-4c3c-8386-2043f72eab6d-maas-api-tls\") pod \"maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:21.238360 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:21.238320 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:21.306164 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:21.305830 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:21.626591 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:21.626564 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr"] Sep 08 10:14:21.627678 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:14:21.627644 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0526bd46_f2e0_4c3c_8386_2043f72eab6d.slice/crio-58bc7347a1ad113c2b0fcb322186597f95de1648a36ce6786c29e92a1fdb14c5 WatchSource:0}: Error finding container 58bc7347a1ad113c2b0fcb322186597f95de1648a36ce6786c29e92a1fdb14c5: Status 404 returned error can't find the container with id 58bc7347a1ad113c2b0fcb322186597f95de1648a36ce6786c29e92a1fdb14c5 Sep 08 10:14:21.708057 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:21.707574 2576 patch_prober.go:28] interesting pod/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 08 10:14:21.708057 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:21.707672 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" podUID="d9ba59c2-f394-4365-84e5-056739f115cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 08 10:14:21.815084 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:21.815016 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:22.631547 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:22.631514 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" event={"ID":"0526bd46-f2e0-4c3c-8386-2043f72eab6d","Type":"ContainerStarted","Data":"d80955a1c9b583620c485e7e9d85391c1767aa445ce2f12cd2edfa3339c23c52"} Sep 08 10:14:22.631945 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:22.631567 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" event={"ID":"0526bd46-f2e0-4c3c-8386-2043f72eab6d","Type":"ContainerStarted","Data":"58bc7347a1ad113c2b0fcb322186597f95de1648a36ce6786c29e92a1fdb14c5"} Sep 08 10:14:22.632227 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:22.632211 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:22.651778 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:22.651729 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" podStartSLOduration=2.3761874 podStartE2EDuration="2.651715614s" totalImagesPullingTime="275.528214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:14:22.650240085 +0000 UTC m=+772.439291890" watchObservedRunningTime="2026-09-08 10:14:22.651715614 +0000 UTC m=+772.440767418" Sep 08 10:14:22.707687 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:22.707645 2576 patch_prober.go:28] interesting pod/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 08 10:14:22.707878 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:22.707712 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" podUID="d9ba59c2-f394-4365-84e5-056739f115cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 08 10:14:22.840119 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:22.840079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:22.941593 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:22.941562 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:23.707632 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:23.707582 2576 patch_prober.go:28] interesting pod/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 08 10:14:23.708113 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:23.707655 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" podUID="d9ba59c2-f394-4365-84e5-056739f115cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 08 10:14:23.785065 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:23.785023 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:24.601983 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:24.601953 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:24.711356 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:24.711326 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:14:24.711804 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:24.711740 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:14:24.712438 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:24.712416 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:14:25.570848 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:25.570812 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:26.883285 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:26.883252 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:28.253560 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:28.253524 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:14:28.631481 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:28.631403 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:28.738610 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:28.738586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:29.456056 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:29.456022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:29.652799 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:29.652768 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:14:39.861667 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:39.861612 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:40.496488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.496428 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s"] Sep 08 10:14:40.497506 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.497452 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" podUID="bad9eb9a-97b9-4599-9c26-aed809766f1d" containerName="maas-api" containerID="cri-o://25dc34dca0fe525b484a9ee8f761dc760d5b6defcbc3f33b280792b2a4e4bc89" gracePeriod=30 Sep 08 10:14:40.534276 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.534246 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:40.619088 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.619058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:40.694412 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.694335 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:40.739563 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.739533 2576 generic.go:356] "Generic (PLEG): container finished" podID="bad9eb9a-97b9-4599-9c26-aed809766f1d" containerID="25dc34dca0fe525b484a9ee8f761dc760d5b6defcbc3f33b280792b2a4e4bc89" exitCode=0 Sep 08 10:14:40.739732 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.739579 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" event={"ID":"bad9eb9a-97b9-4599-9c26-aed809766f1d","Type":"ContainerDied","Data":"25dc34dca0fe525b484a9ee8f761dc760d5b6defcbc3f33b280792b2a4e4bc89"} Sep 08 10:14:40.756867 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.756801 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:14:40.771525 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.771506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:40.917738 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.917700 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad9eb9a-97b9-4599-9c26-aed809766f1d-metrics-certs\") pod \"bad9eb9a-97b9-4599-9c26-aed809766f1d\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " Sep 08 10:14:40.917738 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.917740 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bad9eb9a-97b9-4599-9c26-aed809766f1d-maas-api-tls\") pod \"bad9eb9a-97b9-4599-9c26-aed809766f1d\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " Sep 08 10:14:40.918235 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.917816 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp6nw\" (UniqueName: \"kubernetes.io/projected/bad9eb9a-97b9-4599-9c26-aed809766f1d-kube-api-access-hp6nw\") pod \"bad9eb9a-97b9-4599-9c26-aed809766f1d\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " Sep 08 10:14:40.918235 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.917835 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bad9eb9a-97b9-4599-9c26-aed809766f1d-openshift-service-ca\") pod \"bad9eb9a-97b9-4599-9c26-aed809766f1d\" (UID: \"bad9eb9a-97b9-4599-9c26-aed809766f1d\") " Sep 08 10:14:40.918499 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.918443 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad9eb9a-97b9-4599-9c26-aed809766f1d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bad9eb9a-97b9-4599-9c26-aed809766f1d" (UID: "bad9eb9a-97b9-4599-9c26-aed809766f1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:14:40.919816 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.919789 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:40.920555 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.920526 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad9eb9a-97b9-4599-9c26-aed809766f1d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bad9eb9a-97b9-4599-9c26-aed809766f1d" (UID: "bad9eb9a-97b9-4599-9c26-aed809766f1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:14:40.920669 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.920587 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad9eb9a-97b9-4599-9c26-aed809766f1d-kube-api-access-hp6nw" (OuterVolumeSpecName: "kube-api-access-hp6nw") pod "bad9eb9a-97b9-4599-9c26-aed809766f1d" (UID: "bad9eb9a-97b9-4599-9c26-aed809766f1d"). InnerVolumeSpecName "kube-api-access-hp6nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:14:40.920723 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:40.920666 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad9eb9a-97b9-4599-9c26-aed809766f1d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bad9eb9a-97b9-4599-9c26-aed809766f1d" (UID: "bad9eb9a-97b9-4599-9c26-aed809766f1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:14:41.019382 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:41.019284 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad9eb9a-97b9-4599-9c26-aed809766f1d-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:14:41.019382 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:41.019340 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bad9eb9a-97b9-4599-9c26-aed809766f1d-maas-api-tls\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:14:41.019382 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:41.019356 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp6nw\" (UniqueName: \"kubernetes.io/projected/bad9eb9a-97b9-4599-9c26-aed809766f1d-kube-api-access-hp6nw\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:14:41.019382 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:41.019372 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bad9eb9a-97b9-4599-9c26-aed809766f1d-openshift-service-ca\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:14:41.747358 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:41.747323 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" Sep 08 10:14:41.747358 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:41.747353 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s" event={"ID":"bad9eb9a-97b9-4599-9c26-aed809766f1d","Type":"ContainerDied","Data":"8e61687c869d8f258bd7b97101e8081f01e3260224e8cff5f73ea60e1595fc36"} Sep 08 10:14:41.747561 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:41.747412 2576 scope.go:117] "RemoveContainer" containerID="25dc34dca0fe525b484a9ee8f761dc760d5b6defcbc3f33b280792b2a4e4bc89" Sep 08 10:14:41.772863 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:41.772830 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s"] Sep 08 10:14:41.775209 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:41.775189 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbf4760c-64cfbb69b8-wj54s"] Sep 08 10:14:41.988540 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:41.988511 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l"] Sep 08 10:14:41.989149 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:41.989127 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad9eb9a-97b9-4599-9c26-aed809766f1d" containerName="maas-api" Sep 08 10:14:41.989149 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:41.989150 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad9eb9a-97b9-4599-9c26-aed809766f1d" containerName="maas-api" Sep 08 10:14:41.989323 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:41.989272 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bad9eb9a-97b9-4599-9c26-aed809766f1d" containerName="maas-api" Sep 08 10:14:41.994447 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:41.994424 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:41.998219 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:41.998156 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a5df2a72-gw-openshift-default-dockercfg-77c44\"" Sep 08 10:14:42.019200 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.019177 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l"] Sep 08 10:14:42.027722 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.027696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk9mw\" (UniqueName: \"kubernetes.io/projected/49be8bca-43e4-446d-8bf9-2018e62b2132-kube-api-access-wk9mw\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.027839 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.027738 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-envoy\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.027839 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.027764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-workload-certs\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.027839 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.027786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-token\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.027968 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.027889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-credential-socket\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.027968 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.027926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49be8bca-43e4-446d-8bf9-2018e62b2132-istiod-ca-cert\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.027968 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.027960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-workload-socket\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.028088 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.028073 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-data\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.028135 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.028117 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-podinfo\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.129213 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.129168 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-data\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.129402 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.129228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-podinfo\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.129473 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.129416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk9mw\" (UniqueName: \"kubernetes.io/projected/49be8bca-43e4-446d-8bf9-2018e62b2132-kube-api-access-wk9mw\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.129534 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.129467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-envoy\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.129534 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.129501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-workload-certs\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.129534 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.129527 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-token\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.129726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.129607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-credential-socket\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.129726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.129652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49be8bca-43e4-446d-8bf9-2018e62b2132-istiod-ca-cert\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.129726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.129690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-workload-socket\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.129726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.129713 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-data\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.130043 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.130017 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-credential-socket\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.130203 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.130094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-workload-socket\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.130578 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.130551 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-workload-certs\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.130775 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.130727 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49be8bca-43e4-446d-8bf9-2018e62b2132-istiod-ca-cert\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.132471 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.132448 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-envoy\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.132471 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.132460 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-podinfo\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.147914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.147887 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-token\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.148038 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.148021 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk9mw\" (UniqueName: \"kubernetes.io/projected/49be8bca-43e4-446d-8bf9-2018e62b2132-kube-api-access-wk9mw\") pod \"e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.307742 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.307655 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:42.649910 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:14:42.649881 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49be8bca_43e4_446d_8bf9_2018e62b2132.slice/crio-5494a7b9a34355645cc58bcaf97a25df38723ab062d46f76e4430702620d2aa2 WatchSource:0}: Error finding container 5494a7b9a34355645cc58bcaf97a25df38723ab062d46f76e4430702620d2aa2: Status 404 returned error can't find the container with id 5494a7b9a34355645cc58bcaf97a25df38723ab062d46f76e4430702620d2aa2 Sep 08 10:14:42.652647 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.652588 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:14:42.652716 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.652646 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l"] Sep 08 10:14:42.652716 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.652684 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:14:42.652794 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.652722 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:14:42.679068 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.679040 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bad9eb9a-97b9-4599-9c26-aed809766f1d" path="/var/lib/kubelet/pods/bad9eb9a-97b9-4599-9c26-aed809766f1d/volumes" Sep 08 10:14:42.755209 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.755174 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" event={"ID":"49be8bca-43e4-446d-8bf9-2018e62b2132","Type":"ContainerStarted","Data":"2d815a3b7ddff84637315361e0b02fd5f73e7207f0f7cdaaba96a9a4c02c0332"} Sep 08 10:14:42.755389 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.755236 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" event={"ID":"49be8bca-43e4-446d-8bf9-2018e62b2132","Type":"ContainerStarted","Data":"5494a7b9a34355645cc58bcaf97a25df38723ab062d46f76e4430702620d2aa2"} Sep 08 10:14:42.806771 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:42.806707 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" podStartSLOduration=1.806686443 podStartE2EDuration="1.806686443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:14:42.804315821 +0000 UTC m=+792.593367623" watchObservedRunningTime="2026-09-08 10:14:42.806686443 +0000 UTC m=+792.595738248" Sep 08 10:14:43.308253 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:43.308217 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:43.313508 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:43.313483 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:43.762028 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:43.761996 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:43.762966 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:43.762945 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:14:43.955568 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:43.955538 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:14:44.125511 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.125382 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:44.227195 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.227169 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:44.324985 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.324957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:44.399656 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.399612 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2"] Sep 08 10:14:44.404401 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.404369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:14:44.407232 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.407213 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a5df2a72-serving-cert\"" Sep 08 10:14:44.419189 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.419162 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2"] Sep 08 10:14:44.420335 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.420238 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:44.450352 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.450328 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hvng\" (UniqueName: \"kubernetes.io/projected/07b6fa44-da1b-436f-ae33-79429774449b-kube-api-access-4hvng\") pod \"maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:14:44.450465 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.450383 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b6fa44-da1b-436f-ae33-79429774449b-openshift-service-ca\") pod \"maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:14:44.450465 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.450416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07b6fa44-da1b-436f-ae33-79429774449b-maas-api-tls\") pod \"maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:14:44.450537 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.450466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07b6fa44-da1b-436f-ae33-79429774449b-metrics-certs\") pod \"maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:14:44.551788 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.551748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hvng\" (UniqueName: \"kubernetes.io/projected/07b6fa44-da1b-436f-ae33-79429774449b-kube-api-access-4hvng\") pod \"maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:14:44.551994 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.551842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b6fa44-da1b-436f-ae33-79429774449b-openshift-service-ca\") pod \"maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:14:44.551994 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.551885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07b6fa44-da1b-436f-ae33-79429774449b-maas-api-tls\") pod \"maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:14:44.551994 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.551959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07b6fa44-da1b-436f-ae33-79429774449b-metrics-certs\") pod \"maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:14:44.553295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.553266 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b6fa44-da1b-436f-ae33-79429774449b-openshift-service-ca\") pod \"maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:14:44.555552 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.555526 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07b6fa44-da1b-436f-ae33-79429774449b-metrics-certs\") pod \"maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:14:44.555704 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.555590 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07b6fa44-da1b-436f-ae33-79429774449b-maas-api-tls\") pod \"maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:14:44.565909 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.565882 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hvng\" (UniqueName: \"kubernetes.io/projected/07b6fa44-da1b-436f-ae33-79429774449b-kube-api-access-4hvng\") pod \"maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:14:44.722900 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:44.722827 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:14:45.010070 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:45.010045 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2"] Sep 08 10:14:45.013325 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:14:45.013295 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b6fa44_da1b_436f_ae33_79429774449b.slice/crio-ef07dbe716e1abc7dd6dbf122a32bf31518cd43a40b0ccb51263b899f6bffdf7 WatchSource:0}: Error finding container ef07dbe716e1abc7dd6dbf122a32bf31518cd43a40b0ccb51263b899f6bffdf7: Status 404 returned error can't find the container with id ef07dbe716e1abc7dd6dbf122a32bf31518cd43a40b0ccb51263b899f6bffdf7 Sep 08 10:14:45.777322 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:45.777281 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" event={"ID":"07b6fa44-da1b-436f-ae33-79429774449b","Type":"ContainerStarted","Data":"6b47a11ce39a1b356260bf89767990d37bbffca36702203ad37469fc7ed8fbd3"} Sep 08 10:14:45.777780 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:45.777517 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:14:45.777780 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:45.777553 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" event={"ID":"07b6fa44-da1b-436f-ae33-79429774449b","Type":"ContainerStarted","Data":"ef07dbe716e1abc7dd6dbf122a32bf31518cd43a40b0ccb51263b899f6bffdf7"} Sep 08 10:14:45.797125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:45.797077 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" podStartSLOduration=1.264344929 podStartE2EDuration="1.797058549s" totalImagesPullingTime="532.71362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:14:45.793907798 +0000 UTC m=+795.582959605" watchObservedRunningTime="2026-09-08 10:14:45.797058549 +0000 UTC m=+795.586110380" Sep 08 10:14:46.611514 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:46.611464 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:46.704106 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:46.704071 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:14:51.789195 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:14:51.789165 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:15:03.887140 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:03.887110 2576 generic.go:356] "Generic (PLEG): container finished" podID="49be8bca-43e4-446d-8bf9-2018e62b2132" containerID="2d815a3b7ddff84637315361e0b02fd5f73e7207f0f7cdaaba96a9a4c02c0332" exitCode=0 Sep 08 10:15:03.887467 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:03.887180 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" event={"ID":"49be8bca-43e4-446d-8bf9-2018e62b2132","Type":"ContainerDied","Data":"2d815a3b7ddff84637315361e0b02fd5f73e7207f0f7cdaaba96a9a4c02c0332"} Sep 08 10:15:03.887670 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:03.887652 2576 scope.go:117] "RemoveContainer" containerID="2d815a3b7ddff84637315361e0b02fd5f73e7207f0f7cdaaba96a9a4c02c0332" Sep 08 10:15:03.888736 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:03.888706 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:15:03.888857 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:03.888793 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:15:03.888857 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:03.888849 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:15:04.896710 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:04.896674 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" event={"ID":"49be8bca-43e4-446d-8bf9-2018e62b2132","Type":"ContainerStarted","Data":"8571a46fbe1d8a10d0da041340a3dcdb3d3aa802117bf34cb1e026c9b9de5a73"} Sep 08 10:15:05.308317 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:05.308226 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:15:05.308503 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:05.308482 2576 patch_prober.go:28] interesting pod/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 10:15:05.308579 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:05.308539 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" podUID="49be8bca-43e4-446d-8bf9-2018e62b2132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 10:15:05.553830 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:05.553793 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:15:06.308757 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:06.308714 2576 patch_prober.go:28] interesting pod/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 10:15:06.309380 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:06.309348 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" podUID="49be8bca-43e4-446d-8bf9-2018e62b2132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 10:15:07.309140 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:07.309099 2576 patch_prober.go:28] interesting pod/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 10:15:07.309660 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:07.309181 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" podUID="49be8bca-43e4-446d-8bf9-2018e62b2132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 10:15:08.309004 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:08.308960 2576 patch_prober.go:28] interesting pod/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 10:15:08.309304 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:08.309270 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" podUID="49be8bca-43e4-446d-8bf9-2018e62b2132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 10:15:09.311428 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:09.311393 2576 patch_prober.go:28] interesting pod/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 10:15:09.312012 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:09.311983 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" podUID="49be8bca-43e4-446d-8bf9-2018e62b2132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 10:15:10.308997 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:10.308958 2576 patch_prober.go:28] interesting pod/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 10:15:10.309280 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:10.309249 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" podUID="49be8bca-43e4-446d-8bf9-2018e62b2132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 10:15:11.316718 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:11.316681 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:15:11.317314 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:11.317057 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:15:11.317988 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:11.317970 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:15:14.224039 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:14.224002 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:15.951352 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:15.951316 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2"] Sep 08 10:15:15.951819 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:15.951571 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" podUID="07b6fa44-da1b-436f-ae33-79429774449b" containerName="maas-api" containerID="cri-o://6b47a11ce39a1b356260bf89767990d37bbffca36702203ad37469fc7ed8fbd3" gracePeriod=30 Sep 08 10:15:16.006309 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.006274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:16.087121 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.087088 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:16.187131 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.187104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:16.211138 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.211072 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:15:16.288299 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.288257 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:16.359274 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.359226 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l"] Sep 08 10:15:16.360056 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.359848 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" podUID="49be8bca-43e4-446d-8bf9-2018e62b2132" containerName="istio-proxy" containerID="cri-o://8571a46fbe1d8a10d0da041340a3dcdb3d3aa802117bf34cb1e026c9b9de5a73" gracePeriod=30 Sep 08 10:15:16.366561 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.366539 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:16.379474 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.379456 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hvng\" (UniqueName: \"kubernetes.io/projected/07b6fa44-da1b-436f-ae33-79429774449b-kube-api-access-4hvng\") pod \"07b6fa44-da1b-436f-ae33-79429774449b\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " Sep 08 10:15:16.379597 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.379496 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b6fa44-da1b-436f-ae33-79429774449b-openshift-service-ca\") pod \"07b6fa44-da1b-436f-ae33-79429774449b\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " Sep 08 10:15:16.379597 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.379591 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07b6fa44-da1b-436f-ae33-79429774449b-metrics-certs\") pod \"07b6fa44-da1b-436f-ae33-79429774449b\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " Sep 08 10:15:16.379721 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.379667 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07b6fa44-da1b-436f-ae33-79429774449b-maas-api-tls\") pod \"07b6fa44-da1b-436f-ae33-79429774449b\" (UID: \"07b6fa44-da1b-436f-ae33-79429774449b\") " Sep 08 10:15:16.380271 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.379923 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b6fa44-da1b-436f-ae33-79429774449b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07b6fa44-da1b-436f-ae33-79429774449b" (UID: "07b6fa44-da1b-436f-ae33-79429774449b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:15:16.380271 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.380075 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b6fa44-da1b-436f-ae33-79429774449b-openshift-service-ca\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:16.381750 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.381730 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b6fa44-da1b-436f-ae33-79429774449b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "07b6fa44-da1b-436f-ae33-79429774449b" (UID: "07b6fa44-da1b-436f-ae33-79429774449b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:15:16.381859 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.381771 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b6fa44-da1b-436f-ae33-79429774449b-kube-api-access-4hvng" (OuterVolumeSpecName: "kube-api-access-4hvng") pod "07b6fa44-da1b-436f-ae33-79429774449b" (UID: "07b6fa44-da1b-436f-ae33-79429774449b"). InnerVolumeSpecName "kube-api-access-4hvng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:15:16.382116 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.382098 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b6fa44-da1b-436f-ae33-79429774449b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07b6fa44-da1b-436f-ae33-79429774449b" (UID: "07b6fa44-da1b-436f-ae33-79429774449b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:15:16.481444 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.481362 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07b6fa44-da1b-436f-ae33-79429774449b-maas-api-tls\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:16.481444 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.481391 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hvng\" (UniqueName: \"kubernetes.io/projected/07b6fa44-da1b-436f-ae33-79429774449b-kube-api-access-4hvng\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:16.481444 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.481402 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07b6fa44-da1b-436f-ae33-79429774449b-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:16.839454 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.839365 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:16.990216 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.990177 2576 generic.go:356] "Generic (PLEG): container finished" podID="07b6fa44-da1b-436f-ae33-79429774449b" containerID="6b47a11ce39a1b356260bf89767990d37bbffca36702203ad37469fc7ed8fbd3" exitCode=0 Sep 08 10:15:16.990685 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.990223 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" Sep 08 10:15:16.990685 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.990228 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" event={"ID":"07b6fa44-da1b-436f-ae33-79429774449b","Type":"ContainerDied","Data":"6b47a11ce39a1b356260bf89767990d37bbffca36702203ad37469fc7ed8fbd3"} Sep 08 10:15:16.990685 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.990281 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2" event={"ID":"07b6fa44-da1b-436f-ae33-79429774449b","Type":"ContainerDied","Data":"ef07dbe716e1abc7dd6dbf122a32bf31518cd43a40b0ccb51263b899f6bffdf7"} Sep 08 10:15:16.990685 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:16.990310 2576 scope.go:117] "RemoveContainer" containerID="6b47a11ce39a1b356260bf89767990d37bbffca36702203ad37469fc7ed8fbd3" Sep 08 10:15:17.000081 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:17.000061 2576 scope.go:117] "RemoveContainer" containerID="6b47a11ce39a1b356260bf89767990d37bbffca36702203ad37469fc7ed8fbd3" Sep 08 10:15:17.000342 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:15:17.000321 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b47a11ce39a1b356260bf89767990d37bbffca36702203ad37469fc7ed8fbd3\": container with ID starting with 6b47a11ce39a1b356260bf89767990d37bbffca36702203ad37469fc7ed8fbd3 not found: ID does not exist" containerID="6b47a11ce39a1b356260bf89767990d37bbffca36702203ad37469fc7ed8fbd3" Sep 08 10:15:17.000406 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:17.000353 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b47a11ce39a1b356260bf89767990d37bbffca36702203ad37469fc7ed8fbd3"} err="failed to get container status \"6b47a11ce39a1b356260bf89767990d37bbffca36702203ad37469fc7ed8fbd3\": rpc error: code = NotFound desc = could not find container \"6b47a11ce39a1b356260bf89767990d37bbffca36702203ad37469fc7ed8fbd3\": container with ID starting with 6b47a11ce39a1b356260bf89767990d37bbffca36702203ad37469fc7ed8fbd3 not found: ID does not exist" Sep 08 10:15:17.011301 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:17.011277 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2"] Sep 08 10:15:17.026428 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:17.026402 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5df2a72-67599dbf54-6l9k2"] Sep 08 10:15:17.258097 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:17.258051 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:17.530444 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:17.530367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:18.678522 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:18.678491 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07b6fa44-da1b-436f-ae33-79429774449b" path="/var/lib/kubelet/pods/07b6fa44-da1b-436f-ae33-79429774449b/volumes" Sep 08 10:15:19.770393 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:19.770354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:20.240813 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:20.240772 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:21.088590 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.088561 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr"] Sep 08 10:15:21.089262 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.089239 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07b6fa44-da1b-436f-ae33-79429774449b" containerName="maas-api" Sep 08 10:15:21.089262 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.089262 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="07b6fa44-da1b-436f-ae33-79429774449b" containerName="maas-api" Sep 08 10:15:21.089459 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.089365 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="07b6fa44-da1b-436f-ae33-79429774449b" containerName="maas-api" Sep 08 10:15:21.095096 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.095058 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:21.097365 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.097335 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f688fb08-serving-cert\"" Sep 08 10:15:21.111088 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.111057 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr"] Sep 08 10:15:21.163188 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.163154 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:21.227418 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.227386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400182d5-fe9d-43c2-8ed9-5827a49fa242-maas-api-tls\") pod \"maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:21.227592 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.227429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzrh4\" (UniqueName: \"kubernetes.io/projected/400182d5-fe9d-43c2-8ed9-5827a49fa242-kube-api-access-zzrh4\") pod \"maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:21.227592 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.227574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400182d5-fe9d-43c2-8ed9-5827a49fa242-openshift-service-ca\") pod \"maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:21.227701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.227629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400182d5-fe9d-43c2-8ed9-5827a49fa242-metrics-certs\") pod \"maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:21.265712 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.265684 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:21.328922 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.328887 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400182d5-fe9d-43c2-8ed9-5827a49fa242-metrics-certs\") pod \"maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:21.329072 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.329000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400182d5-fe9d-43c2-8ed9-5827a49fa242-maas-api-tls\") pod \"maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:21.329072 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.329036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzrh4\" (UniqueName: \"kubernetes.io/projected/400182d5-fe9d-43c2-8ed9-5827a49fa242-kube-api-access-zzrh4\") pod \"maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:21.329159 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.329124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400182d5-fe9d-43c2-8ed9-5827a49fa242-openshift-service-ca\") pod \"maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:21.329779 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.329755 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400182d5-fe9d-43c2-8ed9-5827a49fa242-openshift-service-ca\") pod \"maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:21.331533 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.331510 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400182d5-fe9d-43c2-8ed9-5827a49fa242-maas-api-tls\") pod \"maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:21.331652 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.331556 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400182d5-fe9d-43c2-8ed9-5827a49fa242-metrics-certs\") pod \"maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:21.337748 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.337728 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzrh4\" (UniqueName: \"kubernetes.io/projected/400182d5-fe9d-43c2-8ed9-5827a49fa242-kube-api-access-zzrh4\") pod \"maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:21.362637 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.362532 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:21.410913 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.410881 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:21.621661 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.621599 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:15:21.676830 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.676793 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:21.694490 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:15:21.694453 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400182d5_fe9d_43c2_8ed9_5827a49fa242.slice/crio-69bbe8b95cda536b60622f135bd4bde0da9ba7c7e69003d078661eaffa3f010b WatchSource:0}: Error finding container 69bbe8b95cda536b60622f135bd4bde0da9ba7c7e69003d078661eaffa3f010b: Status 404 returned error can't find the container with id 69bbe8b95cda536b60622f135bd4bde0da9ba7c7e69003d078661eaffa3f010b Sep 08 10:15:21.694611 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.694588 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr"] Sep 08 10:15:21.734026 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.733999 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49be8bca-43e4-446d-8bf9-2018e62b2132-istiod-ca-cert\") pod \"49be8bca-43e4-446d-8bf9-2018e62b2132\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " Sep 08 10:15:21.734162 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734040 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-data\") pod \"49be8bca-43e4-446d-8bf9-2018e62b2132\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " Sep 08 10:15:21.734162 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734098 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-workload-socket\") pod \"49be8bca-43e4-446d-8bf9-2018e62b2132\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " Sep 08 10:15:21.734162 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734124 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-token\") pod \"49be8bca-43e4-446d-8bf9-2018e62b2132\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " Sep 08 10:15:21.734349 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734172 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-credential-socket\") pod \"49be8bca-43e4-446d-8bf9-2018e62b2132\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " Sep 08 10:15:21.734349 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734197 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-podinfo\") pod \"49be8bca-43e4-446d-8bf9-2018e62b2132\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " Sep 08 10:15:21.734349 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734221 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-workload-certs\") pod \"49be8bca-43e4-446d-8bf9-2018e62b2132\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " Sep 08 10:15:21.734349 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734340 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-envoy\") pod \"49be8bca-43e4-446d-8bf9-2018e62b2132\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " Sep 08 10:15:21.734545 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734385 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk9mw\" (UniqueName: \"kubernetes.io/projected/49be8bca-43e4-446d-8bf9-2018e62b2132-kube-api-access-wk9mw\") pod \"49be8bca-43e4-446d-8bf9-2018e62b2132\" (UID: \"49be8bca-43e4-446d-8bf9-2018e62b2132\") " Sep 08 10:15:21.734545 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734399 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "49be8bca-43e4-446d-8bf9-2018e62b2132" (UID: "49be8bca-43e4-446d-8bf9-2018e62b2132"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:15:21.734545 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734422 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-data" (OuterVolumeSpecName: "istio-data") pod "49be8bca-43e4-446d-8bf9-2018e62b2132" (UID: "49be8bca-43e4-446d-8bf9-2018e62b2132"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:15:21.734545 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734442 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49be8bca-43e4-446d-8bf9-2018e62b2132-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "49be8bca-43e4-446d-8bf9-2018e62b2132" (UID: "49be8bca-43e4-446d-8bf9-2018e62b2132"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:15:21.734801 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734684 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "49be8bca-43e4-446d-8bf9-2018e62b2132" (UID: "49be8bca-43e4-446d-8bf9-2018e62b2132"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:15:21.734801 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734723 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49be8bca-43e4-446d-8bf9-2018e62b2132-istiod-ca-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:21.734801 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734740 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-data\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:21.734801 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734753 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-workload-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:21.734801 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.734747 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "49be8bca-43e4-446d-8bf9-2018e62b2132" (UID: "49be8bca-43e4-446d-8bf9-2018e62b2132"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:15:21.736467 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.736435 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "49be8bca-43e4-446d-8bf9-2018e62b2132" (UID: "49be8bca-43e4-446d-8bf9-2018e62b2132"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:15:21.736467 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.736450 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-token" (OuterVolumeSpecName: "istio-token") pod "49be8bca-43e4-446d-8bf9-2018e62b2132" (UID: "49be8bca-43e4-446d-8bf9-2018e62b2132"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:15:21.736593 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.736500 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "49be8bca-43e4-446d-8bf9-2018e62b2132" (UID: "49be8bca-43e4-446d-8bf9-2018e62b2132"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 10:15:21.736593 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.736501 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49be8bca-43e4-446d-8bf9-2018e62b2132-kube-api-access-wk9mw" (OuterVolumeSpecName: "kube-api-access-wk9mw") pod "49be8bca-43e4-446d-8bf9-2018e62b2132" (UID: "49be8bca-43e4-446d-8bf9-2018e62b2132"). InnerVolumeSpecName "kube-api-access-wk9mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:15:21.776204 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.776177 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:21.835859 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.835831 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-token\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:21.835859 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.835858 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-credential-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:21.835859 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.835868 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-podinfo\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:21.836097 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.835884 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-workload-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:21.836097 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.835894 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49be8bca-43e4-446d-8bf9-2018e62b2132-istio-envoy\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:21.836097 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:21.835902 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk9mw\" (UniqueName: \"kubernetes.io/projected/49be8bca-43e4-446d-8bf9-2018e62b2132-kube-api-access-wk9mw\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:22.024805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:22.024774 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" event={"ID":"400182d5-fe9d-43c2-8ed9-5827a49fa242","Type":"ContainerStarted","Data":"69bbe8b95cda536b60622f135bd4bde0da9ba7c7e69003d078661eaffa3f010b"} Sep 08 10:15:22.026864 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:22.026822 2576 generic.go:356] "Generic (PLEG): container finished" podID="49be8bca-43e4-446d-8bf9-2018e62b2132" containerID="8571a46fbe1d8a10d0da041340a3dcdb3d3aa802117bf34cb1e026c9b9de5a73" exitCode=0 Sep 08 10:15:22.026988 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:22.026868 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" event={"ID":"49be8bca-43e4-446d-8bf9-2018e62b2132","Type":"ContainerDied","Data":"8571a46fbe1d8a10d0da041340a3dcdb3d3aa802117bf34cb1e026c9b9de5a73"} Sep 08 10:15:22.026988 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:22.026876 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" Sep 08 10:15:22.026988 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:22.026919 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l" event={"ID":"49be8bca-43e4-446d-8bf9-2018e62b2132","Type":"ContainerDied","Data":"5494a7b9a34355645cc58bcaf97a25df38723ab062d46f76e4430702620d2aa2"} Sep 08 10:15:22.026988 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:22.026947 2576 scope.go:117] "RemoveContainer" containerID="8571a46fbe1d8a10d0da041340a3dcdb3d3aa802117bf34cb1e026c9b9de5a73" Sep 08 10:15:22.038218 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:22.038199 2576 scope.go:117] "RemoveContainer" containerID="2d815a3b7ddff84637315361e0b02fd5f73e7207f0f7cdaaba96a9a4c02c0332" Sep 08 10:15:22.050296 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:22.050276 2576 scope.go:117] "RemoveContainer" containerID="8571a46fbe1d8a10d0da041340a3dcdb3d3aa802117bf34cb1e026c9b9de5a73" Sep 08 10:15:22.050592 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:15:22.050570 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8571a46fbe1d8a10d0da041340a3dcdb3d3aa802117bf34cb1e026c9b9de5a73\": container with ID starting with 8571a46fbe1d8a10d0da041340a3dcdb3d3aa802117bf34cb1e026c9b9de5a73 not found: ID does not exist" containerID="8571a46fbe1d8a10d0da041340a3dcdb3d3aa802117bf34cb1e026c9b9de5a73" Sep 08 10:15:22.050714 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:22.050603 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8571a46fbe1d8a10d0da041340a3dcdb3d3aa802117bf34cb1e026c9b9de5a73"} err="failed to get container status \"8571a46fbe1d8a10d0da041340a3dcdb3d3aa802117bf34cb1e026c9b9de5a73\": rpc error: code = NotFound desc = could not find container \"8571a46fbe1d8a10d0da041340a3dcdb3d3aa802117bf34cb1e026c9b9de5a73\": container with ID starting with 8571a46fbe1d8a10d0da041340a3dcdb3d3aa802117bf34cb1e026c9b9de5a73 not found: ID does not exist" Sep 08 10:15:22.050714 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:22.050643 2576 scope.go:117] "RemoveContainer" containerID="2d815a3b7ddff84637315361e0b02fd5f73e7207f0f7cdaaba96a9a4c02c0332" Sep 08 10:15:22.050922 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:15:22.050902 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d815a3b7ddff84637315361e0b02fd5f73e7207f0f7cdaaba96a9a4c02c0332\": container with ID starting with 2d815a3b7ddff84637315361e0b02fd5f73e7207f0f7cdaaba96a9a4c02c0332 not found: ID does not exist" containerID="2d815a3b7ddff84637315361e0b02fd5f73e7207f0f7cdaaba96a9a4c02c0332" Sep 08 10:15:22.050971 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:22.050928 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d815a3b7ddff84637315361e0b02fd5f73e7207f0f7cdaaba96a9a4c02c0332"} err="failed to get container status \"2d815a3b7ddff84637315361e0b02fd5f73e7207f0f7cdaaba96a9a4c02c0332\": rpc error: code = NotFound desc = could not find container \"2d815a3b7ddff84637315361e0b02fd5f73e7207f0f7cdaaba96a9a4c02c0332\": container with ID starting with 2d815a3b7ddff84637315361e0b02fd5f73e7207f0f7cdaaba96a9a4c02c0332 not found: ID does not exist" Sep 08 10:15:22.055094 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:22.055070 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l"] Sep 08 10:15:22.058546 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:22.058517 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a5df2a72-gw-openshift-default-74cdb95576-95r4l"] Sep 08 10:15:22.661669 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:22.661631 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:15:22.678356 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:22.678321 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49be8bca-43e4-446d-8bf9-2018e62b2132" path="/var/lib/kubelet/pods/49be8bca-43e4-446d-8bf9-2018e62b2132/volumes" Sep 08 10:15:23.038673 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:23.038566 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" event={"ID":"400182d5-fe9d-43c2-8ed9-5827a49fa242","Type":"ContainerStarted","Data":"99a585ef22f83bf26ef769a0312540d98bab0f3b330604f5ebbbca85e8412034"} Sep 08 10:15:23.038834 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:23.038681 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:23.059843 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:23.059788 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" podStartSLOduration=1.718160481 podStartE2EDuration="2.059771403s" totalImagesPullingTime="341.610922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:15:23.05755758 +0000 UTC m=+832.846609383" watchObservedRunningTime="2026-09-08 10:15:23.059771403 +0000 UTC m=+832.848823258" Sep 08 10:15:29.050863 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:29.050830 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:33.067738 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:33.067695 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:40.558410 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:40.558374 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:15:48.142217 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:48.142186 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:51.250190 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:51.250151 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:51.744385 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:51.744345 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr"] Sep 08 10:15:51.744604 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:51.744573 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" podUID="400182d5-fe9d-43c2-8ed9-5827a49fa242" containerName="maas-api" containerID="cri-o://99a585ef22f83bf26ef769a0312540d98bab0f3b330604f5ebbbca85e8412034" gracePeriod=30 Sep 08 10:15:51.778755 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:51.778730 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:51.878087 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:51.878041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:52.017989 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.017961 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:52.125462 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.125430 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400182d5-fe9d-43c2-8ed9-5827a49fa242-metrics-certs\") pod \"400182d5-fe9d-43c2-8ed9-5827a49fa242\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " Sep 08 10:15:52.125660 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.125490 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzrh4\" (UniqueName: \"kubernetes.io/projected/400182d5-fe9d-43c2-8ed9-5827a49fa242-kube-api-access-zzrh4\") pod \"400182d5-fe9d-43c2-8ed9-5827a49fa242\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " Sep 08 10:15:52.125660 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.125638 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400182d5-fe9d-43c2-8ed9-5827a49fa242-maas-api-tls\") pod \"400182d5-fe9d-43c2-8ed9-5827a49fa242\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " Sep 08 10:15:52.125801 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.125783 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400182d5-fe9d-43c2-8ed9-5827a49fa242-openshift-service-ca\") pod \"400182d5-fe9d-43c2-8ed9-5827a49fa242\" (UID: \"400182d5-fe9d-43c2-8ed9-5827a49fa242\") " Sep 08 10:15:52.126133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.126111 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400182d5-fe9d-43c2-8ed9-5827a49fa242-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "400182d5-fe9d-43c2-8ed9-5827a49fa242" (UID: "400182d5-fe9d-43c2-8ed9-5827a49fa242"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:15:52.127653 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.127605 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400182d5-fe9d-43c2-8ed9-5827a49fa242-kube-api-access-zzrh4" (OuterVolumeSpecName: "kube-api-access-zzrh4") pod "400182d5-fe9d-43c2-8ed9-5827a49fa242" (UID: "400182d5-fe9d-43c2-8ed9-5827a49fa242"). InnerVolumeSpecName "kube-api-access-zzrh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:15:52.127653 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.127607 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400182d5-fe9d-43c2-8ed9-5827a49fa242-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "400182d5-fe9d-43c2-8ed9-5827a49fa242" (UID: "400182d5-fe9d-43c2-8ed9-5827a49fa242"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:15:52.127784 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.127680 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400182d5-fe9d-43c2-8ed9-5827a49fa242-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "400182d5-fe9d-43c2-8ed9-5827a49fa242" (UID: "400182d5-fe9d-43c2-8ed9-5827a49fa242"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:15:52.221720 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.221683 2576 generic.go:356] "Generic (PLEG): container finished" podID="400182d5-fe9d-43c2-8ed9-5827a49fa242" containerID="99a585ef22f83bf26ef769a0312540d98bab0f3b330604f5ebbbca85e8412034" exitCode=0 Sep 08 10:15:52.221911 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.221728 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" Sep 08 10:15:52.221911 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.221775 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" event={"ID":"400182d5-fe9d-43c2-8ed9-5827a49fa242","Type":"ContainerDied","Data":"99a585ef22f83bf26ef769a0312540d98bab0f3b330604f5ebbbca85e8412034"} Sep 08 10:15:52.221911 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.221838 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr" event={"ID":"400182d5-fe9d-43c2-8ed9-5827a49fa242","Type":"ContainerDied","Data":"69bbe8b95cda536b60622f135bd4bde0da9ba7c7e69003d078661eaffa3f010b"} Sep 08 10:15:52.221911 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.221861 2576 scope.go:117] "RemoveContainer" containerID="99a585ef22f83bf26ef769a0312540d98bab0f3b330604f5ebbbca85e8412034" Sep 08 10:15:52.226917 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.226898 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400182d5-fe9d-43c2-8ed9-5827a49fa242-maas-api-tls\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:52.227027 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.226921 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400182d5-fe9d-43c2-8ed9-5827a49fa242-openshift-service-ca\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:52.227027 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.226934 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400182d5-fe9d-43c2-8ed9-5827a49fa242-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:52.227027 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.226947 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzrh4\" (UniqueName: \"kubernetes.io/projected/400182d5-fe9d-43c2-8ed9-5827a49fa242-kube-api-access-zzrh4\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:15:52.233733 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.233702 2576 scope.go:117] "RemoveContainer" containerID="99a585ef22f83bf26ef769a0312540d98bab0f3b330604f5ebbbca85e8412034" Sep 08 10:15:52.234042 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:15:52.234016 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a585ef22f83bf26ef769a0312540d98bab0f3b330604f5ebbbca85e8412034\": container with ID starting with 99a585ef22f83bf26ef769a0312540d98bab0f3b330604f5ebbbca85e8412034 not found: ID does not exist" containerID="99a585ef22f83bf26ef769a0312540d98bab0f3b330604f5ebbbca85e8412034" Sep 08 10:15:52.234137 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.234050 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a585ef22f83bf26ef769a0312540d98bab0f3b330604f5ebbbca85e8412034"} err="failed to get container status \"99a585ef22f83bf26ef769a0312540d98bab0f3b330604f5ebbbca85e8412034\": rpc error: code = NotFound desc = could not find container \"99a585ef22f83bf26ef769a0312540d98bab0f3b330604f5ebbbca85e8412034\": container with ID starting with 99a585ef22f83bf26ef769a0312540d98bab0f3b330604f5ebbbca85e8412034 not found: ID does not exist" Sep 08 10:15:52.250074 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.250021 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr"] Sep 08 10:15:52.257906 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.257886 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f688fb08-7cbf6f9865-wz8nr"] Sep 08 10:15:52.678336 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:52.678302 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="400182d5-fe9d-43c2-8ed9-5827a49fa242" path="/var/lib/kubelet/pods/400182d5-fe9d-43c2-8ed9-5827a49fa242/volumes" Sep 08 10:15:53.060550 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:53.060472 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:55.473410 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.473352 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx"] Sep 08 10:15:55.474165 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.474143 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49be8bca-43e4-446d-8bf9-2018e62b2132" containerName="istio-proxy" Sep 08 10:15:55.474259 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.474167 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="49be8bca-43e4-446d-8bf9-2018e62b2132" containerName="istio-proxy" Sep 08 10:15:55.474259 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.474179 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400182d5-fe9d-43c2-8ed9-5827a49fa242" containerName="maas-api" Sep 08 10:15:55.474259 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.474187 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="400182d5-fe9d-43c2-8ed9-5827a49fa242" containerName="maas-api" Sep 08 10:15:55.474259 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.474209 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49be8bca-43e4-446d-8bf9-2018e62b2132" containerName="istio-proxy" Sep 08 10:15:55.474259 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.474218 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="49be8bca-43e4-446d-8bf9-2018e62b2132" containerName="istio-proxy" Sep 08 10:15:55.474511 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.474317 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="49be8bca-43e4-446d-8bf9-2018e62b2132" containerName="istio-proxy" Sep 08 10:15:55.474511 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.474332 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="400182d5-fe9d-43c2-8ed9-5827a49fa242" containerName="maas-api" Sep 08 10:15:55.474511 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.474343 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="49be8bca-43e4-446d-8bf9-2018e62b2132" containerName="istio-proxy" Sep 08 10:15:55.479083 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.479061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.481304 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.481281 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-057291ac-openshift-default-dockercfg-6gclv\"" Sep 08 10:15:55.492680 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.492650 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx"] Sep 08 10:15:55.661117 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.661084 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-data\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.661312 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.661137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istiod-ca-cert\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.661312 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.661159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-workload-certs\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.661312 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.661218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-token\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.661312 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.661258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-envoy\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.661312 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.661278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-workload-socket\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.661312 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.661301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch4wv\" (UniqueName: \"kubernetes.io/projected/3250d43d-7f11-46bd-8d58-0c29ed51fc44-kube-api-access-ch4wv\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.661656 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.661382 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-credential-socket\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.661656 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.661420 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-podinfo\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.762416 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.762337 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-credential-socket\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.762416 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.762374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-podinfo\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.762416 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.762412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-data\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.762800 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.762444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istiod-ca-cert\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.762800 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.762463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-workload-certs\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.762800 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.762483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-token\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.762800 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.762552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-envoy\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.762800 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.762595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-workload-socket\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.763055 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.762825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch4wv\" (UniqueName: \"kubernetes.io/projected/3250d43d-7f11-46bd-8d58-0c29ed51fc44-kube-api-access-ch4wv\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.763055 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.762829 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-credential-socket\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.763055 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.762927 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-data\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.763055 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.762947 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-workload-certs\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.763216 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.763133 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-workload-socket\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.763216 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.763200 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istiod-ca-cert\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.764965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.764941 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-envoy\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.765078 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.764978 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-podinfo\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.774150 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.774127 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch4wv\" (UniqueName: \"kubernetes.io/projected/3250d43d-7f11-46bd-8d58-0c29ed51fc44-kube-api-access-ch4wv\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.776191 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.776172 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-token\") pod \"e2e-derive-057291ac-openshift-default-59f858b896-q9jsx\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:55.790900 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:55.790879 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:56.072520 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:56.072493 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx"] Sep 08 10:15:56.073287 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:15:56.073265 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3250d43d_7f11_46bd_8d58_0c29ed51fc44.slice/crio-0d397e20cba238354cfffccec92d9467a149bd76b984fa9723dc570f655249c2 WatchSource:0}: Error finding container 0d397e20cba238354cfffccec92d9467a149bd76b984fa9723dc570f655249c2: Status 404 returned error can't find the container with id 0d397e20cba238354cfffccec92d9467a149bd76b984fa9723dc570f655249c2 Sep 08 10:15:56.075446 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:56.075415 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:15:56.075498 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:56.075482 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:15:56.075533 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:56.075509 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:15:56.124511 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:56.124486 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:56.227417 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:56.227366 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:56.251414 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:56.251379 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" event={"ID":"3250d43d-7f11-46bd-8d58-0c29ed51fc44","Type":"ContainerStarted","Data":"c6a574e5b4e390ab4866b376daf7e4871f658f65e7b79626711c0bd7bda7f0cb"} Sep 08 10:15:56.251588 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:56.251434 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" event={"ID":"3250d43d-7f11-46bd-8d58-0c29ed51fc44","Type":"ContainerStarted","Data":"0d397e20cba238354cfffccec92d9467a149bd76b984fa9723dc570f655249c2"} Sep 08 10:15:56.275430 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:56.275372 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" podStartSLOduration=1.275352082 podStartE2EDuration="1.275352082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:15:56.27179556 +0000 UTC m=+866.060847366" watchObservedRunningTime="2026-09-08 10:15:56.275352082 +0000 UTC m=+866.064403887" Sep 08 10:15:56.791822 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:56.791780 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:57.720805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:57.720769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:57.797405 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:57.797374 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:57.816532 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:57.816510 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:58.262980 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:58.262950 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:58.263963 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:58.263942 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:15:58.840944 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:58.840913 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z"] Sep 08 10:15:58.844577 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:58.844554 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:15:58.846920 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:58.846899 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-057291ac-serving-cert\"" Sep 08 10:15:58.856468 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:58.856447 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z"] Sep 08 10:15:58.891552 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:58.891523 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:58.955187 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:58.955155 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:58.995822 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:58.995741 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe67257e-980c-42f0-b627-dabd8532d2b2-openshift-service-ca\") pod \"maas-api-e2e-derive-057291ac-fd8c4c96-6k49z\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:15:58.995822 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:58.995811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe67257e-980c-42f0-b627-dabd8532d2b2-metrics-certs\") pod \"maas-api-e2e-derive-057291ac-fd8c4c96-6k49z\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:15:58.996072 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:58.995841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f5fn\" (UniqueName: \"kubernetes.io/projected/fe67257e-980c-42f0-b627-dabd8532d2b2-kube-api-access-7f5fn\") pod \"maas-api-e2e-derive-057291ac-fd8c4c96-6k49z\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:15:58.996072 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:58.995888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe67257e-980c-42f0-b627-dabd8532d2b2-maas-api-tls\") pod \"maas-api-e2e-derive-057291ac-fd8c4c96-6k49z\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:15:59.097399 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:59.097311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe67257e-980c-42f0-b627-dabd8532d2b2-maas-api-tls\") pod \"maas-api-e2e-derive-057291ac-fd8c4c96-6k49z\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:15:59.097560 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:59.097430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe67257e-980c-42f0-b627-dabd8532d2b2-openshift-service-ca\") pod \"maas-api-e2e-derive-057291ac-fd8c4c96-6k49z\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:15:59.097560 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:59.097459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe67257e-980c-42f0-b627-dabd8532d2b2-metrics-certs\") pod \"maas-api-e2e-derive-057291ac-fd8c4c96-6k49z\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:15:59.097560 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:59.097476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f5fn\" (UniqueName: \"kubernetes.io/projected/fe67257e-980c-42f0-b627-dabd8532d2b2-kube-api-access-7f5fn\") pod \"maas-api-e2e-derive-057291ac-fd8c4c96-6k49z\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:15:59.097560 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:15:59.097483 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-serving-cert: secret "maas-api-e2e-derive-057291ac-serving-cert" not found Sep 08 10:15:59.097560 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:15:59.097558 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe67257e-980c-42f0-b627-dabd8532d2b2-maas-api-tls podName:fe67257e-980c-42f0-b627-dabd8532d2b2 nodeName:}" failed. No retries permitted until 2026-09-08 10:15:59.597535018 +0000 UTC m=+869.386586816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe67257e-980c-42f0-b627-dabd8532d2b2-maas-api-tls") pod "maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" (UID: "fe67257e-980c-42f0-b627-dabd8532d2b2") : secret "maas-api-e2e-derive-057291ac-serving-cert" not found Sep 08 10:15:59.098159 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:59.098133 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe67257e-980c-42f0-b627-dabd8532d2b2-openshift-service-ca\") pod \"maas-api-e2e-derive-057291ac-fd8c4c96-6k49z\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:15:59.100136 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:59.100113 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe67257e-980c-42f0-b627-dabd8532d2b2-metrics-certs\") pod \"maas-api-e2e-derive-057291ac-fd8c4c96-6k49z\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:15:59.106264 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:59.106238 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f5fn\" (UniqueName: \"kubernetes.io/projected/fe67257e-980c-42f0-b627-dabd8532d2b2-kube-api-access-7f5fn\") pod \"maas-api-e2e-derive-057291ac-fd8c4c96-6k49z\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:15:59.242590 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:59.242555 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:59.408307 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:59.408280 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:59.505533 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:59.505496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:15:59.603275 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:59.603231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe67257e-980c-42f0-b627-dabd8532d2b2-maas-api-tls\") pod \"maas-api-e2e-derive-057291ac-fd8c4c96-6k49z\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:15:59.605776 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:59.605751 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe67257e-980c-42f0-b627-dabd8532d2b2-maas-api-tls\") pod \"maas-api-e2e-derive-057291ac-fd8c4c96-6k49z\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:15:59.757997 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:15:59.757911 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:16:00.041046 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:00.041017 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z"] Sep 08 10:16:00.042195 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:16:00.042159 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe67257e_980c_42f0_b627_dabd8532d2b2.slice/crio-c5052526e1940530fb4e4332ee198008a7f1520932d2956cc4fc707e76a5ea8c WatchSource:0}: Error finding container c5052526e1940530fb4e4332ee198008a7f1520932d2956cc4fc707e76a5ea8c: Status 404 returned error can't find the container with id c5052526e1940530fb4e4332ee198008a7f1520932d2956cc4fc707e76a5ea8c Sep 08 10:16:00.275919 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:00.275886 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" event={"ID":"fe67257e-980c-42f0-b627-dabd8532d2b2","Type":"ContainerStarted","Data":"c5052526e1940530fb4e4332ee198008a7f1520932d2956cc4fc707e76a5ea8c"} Sep 08 10:16:01.289729 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:01.289680 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" event={"ID":"fe67257e-980c-42f0-b627-dabd8532d2b2","Type":"ContainerStarted","Data":"dc64cc5f374d923b6128ac1623178edd7193cb5a6cc06a01821e9078c28ad30e"} Sep 08 10:16:01.290221 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:01.289780 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:16:01.309571 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:01.309514 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" podStartSLOduration=2.689159072 podStartE2EDuration="3.309493234s" totalImagesPullingTime="620.334162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:16:01.30660543 +0000 UTC m=+871.095657227" watchObservedRunningTime="2026-09-08 10:16:01.309493234 +0000 UTC m=+871.098545038" Sep 08 10:16:03.461175 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:03.461136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:04.114221 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:04.114186 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:04.212850 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:04.212808 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:04.291141 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:04.291098 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:04.378953 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:04.378871 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:04.440304 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:04.440269 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:05.022542 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:05.022506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:05.060778 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:05.060746 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:16:07.309194 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:07.309158 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:16:10.913010 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:10.912948 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:11.158566 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:11.158534 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr"] Sep 08 10:16:11.158832 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:11.158789 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" podUID="d9ba59c2-f394-4365-84e5-056739f115cc" containerName="istio-proxy" containerID="cri-o://fdd36be16f4586f1780e2d2f7c18f31727b636881a6cc7702b568b8a15460861" gracePeriod=30 Sep 08 10:16:13.066797 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:13.066763 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:13.147251 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:13.147213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:14.303765 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:14.303728 2576 patch_prober.go:28] interesting pod/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 10:16:14.304127 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:14.303785 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" podUID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 10:16:14.651469 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:14.651433 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:14.745579 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:14.745541 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:15.796986 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:15.796956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:15.907679 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:15.907646 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:16.383836 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.383805 2576 generic.go:356] "Generic (PLEG): container finished" podID="d9ba59c2-f394-4365-84e5-056739f115cc" containerID="fdd36be16f4586f1780e2d2f7c18f31727b636881a6cc7702b568b8a15460861" exitCode=0 Sep 08 10:16:16.384017 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.383879 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" event={"ID":"d9ba59c2-f394-4365-84e5-056739f115cc","Type":"ContainerDied","Data":"fdd36be16f4586f1780e2d2f7c18f31727b636881a6cc7702b568b8a15460861"} Sep 08 10:16:16.384017 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.383946 2576 scope.go:117] "RemoveContainer" containerID="3c6d3e26d3a28b4d29ad71c8865de9496ac6505760923ae4348ded34482447ed" Sep 08 10:16:16.385694 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.385674 2576 generic.go:356] "Generic (PLEG): container finished" podID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerID="c6a574e5b4e390ab4866b376daf7e4871f658f65e7b79626711c0bd7bda7f0cb" exitCode=0 Sep 08 10:16:16.385808 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.385740 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" event={"ID":"3250d43d-7f11-46bd-8d58-0c29ed51fc44","Type":"ContainerDied","Data":"c6a574e5b4e390ab4866b376daf7e4871f658f65e7b79626711c0bd7bda7f0cb"} Sep 08 10:16:16.386145 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.386131 2576 scope.go:117] "RemoveContainer" containerID="c6a574e5b4e390ab4866b376daf7e4871f658f65e7b79626711c0bd7bda7f0cb" Sep 08 10:16:16.387496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.387466 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:16:16.387608 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.387540 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:16:16.387608 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.387574 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:16:16.446633 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.446587 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:16:16.486701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.485694 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-istio-envoy\") pod \"d9ba59c2-f394-4365-84e5-056739f115cc\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " Sep 08 10:16:16.486701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.485742 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ba59c2-f394-4365-84e5-056739f115cc-istiod-ca-cert\") pod \"d9ba59c2-f394-4365-84e5-056739f115cc\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " Sep 08 10:16:16.486701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.485793 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-istio-data\") pod \"d9ba59c2-f394-4365-84e5-056739f115cc\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " Sep 08 10:16:16.486701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.485820 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-workload-socket\") pod \"d9ba59c2-f394-4365-84e5-056739f115cc\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " Sep 08 10:16:16.486701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.485854 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-workload-certs\") pod \"d9ba59c2-f394-4365-84e5-056739f115cc\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " Sep 08 10:16:16.486701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.485898 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzzzn\" (UniqueName: \"kubernetes.io/projected/d9ba59c2-f394-4365-84e5-056739f115cc-kube-api-access-kzzzn\") pod \"d9ba59c2-f394-4365-84e5-056739f115cc\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " Sep 08 10:16:16.486701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.485942 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-credential-socket\") pod \"d9ba59c2-f394-4365-84e5-056739f115cc\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " Sep 08 10:16:16.486701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.485978 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ba59c2-f394-4365-84e5-056739f115cc-istio-token\") pod \"d9ba59c2-f394-4365-84e5-056739f115cc\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " Sep 08 10:16:16.486701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.486000 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ba59c2-f394-4365-84e5-056739f115cc-istio-podinfo\") pod \"d9ba59c2-f394-4365-84e5-056739f115cc\" (UID: \"d9ba59c2-f394-4365-84e5-056739f115cc\") " Sep 08 10:16:16.486701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.486289 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9ba59c2-f394-4365-84e5-056739f115cc" (UID: "d9ba59c2-f394-4365-84e5-056739f115cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:16:16.486701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.486379 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ba59c2-f394-4365-84e5-056739f115cc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9ba59c2-f394-4365-84e5-056739f115cc" (UID: "d9ba59c2-f394-4365-84e5-056739f115cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:16:16.486701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.486493 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ba59c2-f394-4365-84e5-056739f115cc-istiod-ca-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:16.486701 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.486511 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-workload-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:16.487476 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.486951 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9ba59c2-f394-4365-84e5-056739f115cc" (UID: "d9ba59c2-f394-4365-84e5-056739f115cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:16:16.487476 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.487303 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9ba59c2-f394-4365-84e5-056739f115cc" (UID: "d9ba59c2-f394-4365-84e5-056739f115cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:16:16.487902 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.487820 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9ba59c2-f394-4365-84e5-056739f115cc" (UID: "d9ba59c2-f394-4365-84e5-056739f115cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:16:16.489670 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.489611 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ba59c2-f394-4365-84e5-056739f115cc-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9ba59c2-f394-4365-84e5-056739f115cc" (UID: "d9ba59c2-f394-4365-84e5-056739f115cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:16:16.490194 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.490167 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9ba59c2-f394-4365-84e5-056739f115cc" (UID: "d9ba59c2-f394-4365-84e5-056739f115cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:16:16.493736 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.493697 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9ba59c2-f394-4365-84e5-056739f115cc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9ba59c2-f394-4365-84e5-056739f115cc" (UID: "d9ba59c2-f394-4365-84e5-056739f115cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 10:16:16.493940 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.493921 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ba59c2-f394-4365-84e5-056739f115cc-kube-api-access-kzzzn" (OuterVolumeSpecName: "kube-api-access-kzzzn") pod "d9ba59c2-f394-4365-84e5-056739f115cc" (UID: "d9ba59c2-f394-4365-84e5-056739f115cc"). InnerVolumeSpecName "kube-api-access-kzzzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:16:16.587406 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.587368 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-workload-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:16.587552 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.587410 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzzzn\" (UniqueName: \"kubernetes.io/projected/d9ba59c2-f394-4365-84e5-056739f115cc-kube-api-access-kzzzn\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:16.587552 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.587425 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-credential-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:16.587552 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.587438 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ba59c2-f394-4365-84e5-056739f115cc-istio-token\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:16.587552 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.587451 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ba59c2-f394-4365-84e5-056739f115cc-istio-podinfo\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:16.587552 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.587465 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-istio-envoy\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:16.587552 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:16.587477 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ba59c2-f394-4365-84e5-056739f115cc-istio-data\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:17.396191 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:17.396143 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" event={"ID":"3250d43d-7f11-46bd-8d58-0c29ed51fc44","Type":"ContainerStarted","Data":"07cc488281197427d2d62f22860053571bbb859d78045235187969479cfdd808"} Sep 08 10:16:17.401687 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:17.401650 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" event={"ID":"d9ba59c2-f394-4365-84e5-056739f115cc","Type":"ContainerDied","Data":"033e55bf235df91b41a3465a9f91f5aeafe427760a04277dea8d48d43bf052fb"} Sep 08 10:16:17.402061 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:17.402041 2576 scope.go:117] "RemoveContainer" containerID="fdd36be16f4586f1780e2d2f7c18f31727b636881a6cc7702b568b8a15460861" Sep 08 10:16:17.402275 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:17.401975 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr" Sep 08 10:16:17.443887 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:17.443845 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr"] Sep 08 10:16:17.448186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:17.448159 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-338aba-openshift-default-6ffd89c954-hjzqr"] Sep 08 10:16:17.792217 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:17.792106 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:16:17.792409 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:17.792306 2576 patch_prober.go:28] interesting pod/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 10:16:17.792409 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:17.792359 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" podUID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 10:16:17.954999 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:17.954965 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:18.680188 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:18.680155 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9ba59c2-f394-4365-84e5-056739f115cc" path="/var/lib/kubelet/pods/d9ba59c2-f394-4365-84e5-056739f115cc/volumes" Sep 08 10:16:18.791377 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:18.791339 2576 patch_prober.go:28] interesting pod/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 10:16:18.791686 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:18.791655 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" podUID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 10:16:19.791534 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:19.791497 2576 patch_prober.go:28] interesting pod/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 10:16:19.792134 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:19.792098 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" podUID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 10:16:20.117128 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:20.117013 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:20.217203 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:20.217159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:20.792058 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:20.792014 2576 patch_prober.go:28] interesting pod/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 10:16:20.792573 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:20.792087 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" podUID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 10:16:20.982496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:20.982461 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:21.696166 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:21.696128 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:21.791716 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:21.791676 2576 patch_prober.go:28] interesting pod/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 10:16:21.792036 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:21.792005 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" podUID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 10:16:22.792000 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:22.791966 2576 patch_prober.go:28] interesting pod/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 10:16:22.792487 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:22.792025 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" podUID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 10:16:23.796751 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:23.796719 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:16:23.797212 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:23.797098 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:16:23.797947 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:23.797927 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:16:27.649490 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:27.649462 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:29.349426 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:29.349386 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:16:29.606607 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:29.606522 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z"] Sep 08 10:16:29.606780 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:29.606758 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" podUID="fe67257e-980c-42f0-b627-dabd8532d2b2" containerName="maas-api" containerID="cri-o://dc64cc5f374d923b6128ac1623178edd7193cb5a6cc06a01821e9078c28ad30e" gracePeriod=30 Sep 08 10:16:29.869514 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:29.869454 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:16:29.986500 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:29.986467 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx"] Sep 08 10:16:29.986713 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:29.986690 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" podUID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerName="istio-proxy" containerID="cri-o://07cc488281197427d2d62f22860053571bbb859d78045235187969479cfdd808" gracePeriod=30 Sep 08 10:16:30.035275 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.035227 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe67257e-980c-42f0-b627-dabd8532d2b2-metrics-certs\") pod \"fe67257e-980c-42f0-b627-dabd8532d2b2\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " Sep 08 10:16:30.035444 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.035338 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe67257e-980c-42f0-b627-dabd8532d2b2-openshift-service-ca\") pod \"fe67257e-980c-42f0-b627-dabd8532d2b2\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " Sep 08 10:16:30.035444 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.035405 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f5fn\" (UniqueName: \"kubernetes.io/projected/fe67257e-980c-42f0-b627-dabd8532d2b2-kube-api-access-7f5fn\") pod \"fe67257e-980c-42f0-b627-dabd8532d2b2\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " Sep 08 10:16:30.035538 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.035453 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe67257e-980c-42f0-b627-dabd8532d2b2-maas-api-tls\") pod \"fe67257e-980c-42f0-b627-dabd8532d2b2\" (UID: \"fe67257e-980c-42f0-b627-dabd8532d2b2\") " Sep 08 10:16:30.035779 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.035753 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe67257e-980c-42f0-b627-dabd8532d2b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe67257e-980c-42f0-b627-dabd8532d2b2" (UID: "fe67257e-980c-42f0-b627-dabd8532d2b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:16:30.037350 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.037319 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe67257e-980c-42f0-b627-dabd8532d2b2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fe67257e-980c-42f0-b627-dabd8532d2b2" (UID: "fe67257e-980c-42f0-b627-dabd8532d2b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:16:30.037547 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.037511 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe67257e-980c-42f0-b627-dabd8532d2b2-kube-api-access-7f5fn" (OuterVolumeSpecName: "kube-api-access-7f5fn") pod "fe67257e-980c-42f0-b627-dabd8532d2b2" (UID: "fe67257e-980c-42f0-b627-dabd8532d2b2"). InnerVolumeSpecName "kube-api-access-7f5fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:16:30.037597 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.037521 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe67257e-980c-42f0-b627-dabd8532d2b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe67257e-980c-42f0-b627-dabd8532d2b2" (UID: "fe67257e-980c-42f0-b627-dabd8532d2b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:16:30.136858 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.136763 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f5fn\" (UniqueName: \"kubernetes.io/projected/fe67257e-980c-42f0-b627-dabd8532d2b2-kube-api-access-7f5fn\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:30.136858 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.136798 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe67257e-980c-42f0-b627-dabd8532d2b2-maas-api-tls\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:30.136858 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.136807 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe67257e-980c-42f0-b627-dabd8532d2b2-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:30.136858 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.136815 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe67257e-980c-42f0-b627-dabd8532d2b2-openshift-service-ca\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:30.487868 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.487825 2576 generic.go:356] "Generic (PLEG): container finished" podID="fe67257e-980c-42f0-b627-dabd8532d2b2" containerID="dc64cc5f374d923b6128ac1623178edd7193cb5a6cc06a01821e9078c28ad30e" exitCode=0 Sep 08 10:16:30.488336 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.487884 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" Sep 08 10:16:30.488336 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.487886 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" event={"ID":"fe67257e-980c-42f0-b627-dabd8532d2b2","Type":"ContainerDied","Data":"dc64cc5f374d923b6128ac1623178edd7193cb5a6cc06a01821e9078c28ad30e"} Sep 08 10:16:30.488336 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.488007 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z" event={"ID":"fe67257e-980c-42f0-b627-dabd8532d2b2","Type":"ContainerDied","Data":"c5052526e1940530fb4e4332ee198008a7f1520932d2956cc4fc707e76a5ea8c"} Sep 08 10:16:30.488336 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.488090 2576 scope.go:117] "RemoveContainer" containerID="dc64cc5f374d923b6128ac1623178edd7193cb5a6cc06a01821e9078c28ad30e" Sep 08 10:16:30.499437 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.499414 2576 scope.go:117] "RemoveContainer" containerID="dc64cc5f374d923b6128ac1623178edd7193cb5a6cc06a01821e9078c28ad30e" Sep 08 10:16:30.499839 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:16:30.499810 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc64cc5f374d923b6128ac1623178edd7193cb5a6cc06a01821e9078c28ad30e\": container with ID starting with dc64cc5f374d923b6128ac1623178edd7193cb5a6cc06a01821e9078c28ad30e not found: ID does not exist" containerID="dc64cc5f374d923b6128ac1623178edd7193cb5a6cc06a01821e9078c28ad30e" Sep 08 10:16:30.499945 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.499851 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc64cc5f374d923b6128ac1623178edd7193cb5a6cc06a01821e9078c28ad30e"} err="failed to get container status \"dc64cc5f374d923b6128ac1623178edd7193cb5a6cc06a01821e9078c28ad30e\": rpc error: code = NotFound desc = could not find container \"dc64cc5f374d923b6128ac1623178edd7193cb5a6cc06a01821e9078c28ad30e\": container with ID starting with dc64cc5f374d923b6128ac1623178edd7193cb5a6cc06a01821e9078c28ad30e not found: ID does not exist" Sep 08 10:16:30.512452 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.512425 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z"] Sep 08 10:16:30.518803 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.518772 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-057291ac-fd8c4c96-6k49z"] Sep 08 10:16:30.680608 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:30.680571 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe67257e-980c-42f0-b627-dabd8532d2b2" path="/var/lib/kubelet/pods/fe67257e-980c-42f0-b627-dabd8532d2b2/volumes" Sep 08 10:16:31.302847 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:31.302765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:31.403101 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:31.403066 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:34.085032 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:34.084998 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:35.239611 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.239581 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:16:35.390989 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.390955 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-workload-socket\") pod \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " Sep 08 10:16:35.391174 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.391005 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch4wv\" (UniqueName: \"kubernetes.io/projected/3250d43d-7f11-46bd-8d58-0c29ed51fc44-kube-api-access-ch4wv\") pod \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " Sep 08 10:16:35.391174 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.391057 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-podinfo\") pod \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " Sep 08 10:16:35.391174 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.391086 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-envoy\") pod \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " Sep 08 10:16:35.391340 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.391202 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-token\") pod \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " Sep 08 10:16:35.391340 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.391255 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-workload-certs\") pod \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " Sep 08 10:16:35.391340 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.391266 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3250d43d-7f11-46bd-8d58-0c29ed51fc44" (UID: "3250d43d-7f11-46bd-8d58-0c29ed51fc44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:16:35.391340 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.391315 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istiod-ca-cert\") pod \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " Sep 08 10:16:35.391536 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.391356 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-credential-socket\") pod \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " Sep 08 10:16:35.391536 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.391402 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-data\") pod \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\" (UID: \"3250d43d-7f11-46bd-8d58-0c29ed51fc44\") " Sep 08 10:16:35.392000 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.391771 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-workload-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:35.392000 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.391790 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3250d43d-7f11-46bd-8d58-0c29ed51fc44" (UID: "3250d43d-7f11-46bd-8d58-0c29ed51fc44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:16:35.392000 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.391973 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-data" (OuterVolumeSpecName: "istio-data") pod "3250d43d-7f11-46bd-8d58-0c29ed51fc44" (UID: "3250d43d-7f11-46bd-8d58-0c29ed51fc44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:16:35.392274 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.392005 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3250d43d-7f11-46bd-8d58-0c29ed51fc44" (UID: "3250d43d-7f11-46bd-8d58-0c29ed51fc44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:16:35.392274 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.392053 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3250d43d-7f11-46bd-8d58-0c29ed51fc44" (UID: "3250d43d-7f11-46bd-8d58-0c29ed51fc44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:16:35.393533 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.393508 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3250d43d-7f11-46bd-8d58-0c29ed51fc44" (UID: "3250d43d-7f11-46bd-8d58-0c29ed51fc44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 10:16:35.393730 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.393699 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3250d43d-7f11-46bd-8d58-0c29ed51fc44" (UID: "3250d43d-7f11-46bd-8d58-0c29ed51fc44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:16:35.393730 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.393714 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-token" (OuterVolumeSpecName: "istio-token") pod "3250d43d-7f11-46bd-8d58-0c29ed51fc44" (UID: "3250d43d-7f11-46bd-8d58-0c29ed51fc44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:16:35.393913 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.393894 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3250d43d-7f11-46bd-8d58-0c29ed51fc44-kube-api-access-ch4wv" (OuterVolumeSpecName: "kube-api-access-ch4wv") pod "3250d43d-7f11-46bd-8d58-0c29ed51fc44" (UID: "3250d43d-7f11-46bd-8d58-0c29ed51fc44"). InnerVolumeSpecName "kube-api-access-ch4wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:16:35.493375 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.493337 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-token\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:35.493375 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.493369 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-workload-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:35.493375 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.493379 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istiod-ca-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:35.493695 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.493387 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-credential-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:35.493695 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.493397 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-data\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:35.493695 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.493405 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch4wv\" (UniqueName: \"kubernetes.io/projected/3250d43d-7f11-46bd-8d58-0c29ed51fc44-kube-api-access-ch4wv\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:35.493695 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.493418 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-podinfo\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:35.493695 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.493429 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3250d43d-7f11-46bd-8d58-0c29ed51fc44-istio-envoy\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:16:35.521720 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.521682 2576 generic.go:356] "Generic (PLEG): container finished" podID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerID="07cc488281197427d2d62f22860053571bbb859d78045235187969479cfdd808" exitCode=0 Sep 08 10:16:35.521898 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.521752 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" Sep 08 10:16:35.521898 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.521769 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" event={"ID":"3250d43d-7f11-46bd-8d58-0c29ed51fc44","Type":"ContainerDied","Data":"07cc488281197427d2d62f22860053571bbb859d78045235187969479cfdd808"} Sep 08 10:16:35.521898 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.521829 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx" event={"ID":"3250d43d-7f11-46bd-8d58-0c29ed51fc44","Type":"ContainerDied","Data":"0d397e20cba238354cfffccec92d9467a149bd76b984fa9723dc570f655249c2"} Sep 08 10:16:35.521898 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.521852 2576 scope.go:117] "RemoveContainer" containerID="07cc488281197427d2d62f22860053571bbb859d78045235187969479cfdd808" Sep 08 10:16:35.534298 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.534282 2576 scope.go:117] "RemoveContainer" containerID="c6a574e5b4e390ab4866b376daf7e4871f658f65e7b79626711c0bd7bda7f0cb" Sep 08 10:16:35.543332 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.543307 2576 scope.go:117] "RemoveContainer" containerID="07cc488281197427d2d62f22860053571bbb859d78045235187969479cfdd808" Sep 08 10:16:35.543597 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:16:35.543578 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07cc488281197427d2d62f22860053571bbb859d78045235187969479cfdd808\": container with ID starting with 07cc488281197427d2d62f22860053571bbb859d78045235187969479cfdd808 not found: ID does not exist" containerID="07cc488281197427d2d62f22860053571bbb859d78045235187969479cfdd808" Sep 08 10:16:35.543691 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.543605 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07cc488281197427d2d62f22860053571bbb859d78045235187969479cfdd808"} err="failed to get container status \"07cc488281197427d2d62f22860053571bbb859d78045235187969479cfdd808\": rpc error: code = NotFound desc = could not find container \"07cc488281197427d2d62f22860053571bbb859d78045235187969479cfdd808\": container with ID starting with 07cc488281197427d2d62f22860053571bbb859d78045235187969479cfdd808 not found: ID does not exist" Sep 08 10:16:35.543691 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.543643 2576 scope.go:117] "RemoveContainer" containerID="c6a574e5b4e390ab4866b376daf7e4871f658f65e7b79626711c0bd7bda7f0cb" Sep 08 10:16:35.543917 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:16:35.543899 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a574e5b4e390ab4866b376daf7e4871f658f65e7b79626711c0bd7bda7f0cb\": container with ID starting with c6a574e5b4e390ab4866b376daf7e4871f658f65e7b79626711c0bd7bda7f0cb not found: ID does not exist" containerID="c6a574e5b4e390ab4866b376daf7e4871f658f65e7b79626711c0bd7bda7f0cb" Sep 08 10:16:35.543974 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.543927 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a574e5b4e390ab4866b376daf7e4871f658f65e7b79626711c0bd7bda7f0cb"} err="failed to get container status \"c6a574e5b4e390ab4866b376daf7e4871f658f65e7b79626711c0bd7bda7f0cb\": rpc error: code = NotFound desc = could not find container \"c6a574e5b4e390ab4866b376daf7e4871f658f65e7b79626711c0bd7bda7f0cb\": container with ID starting with c6a574e5b4e390ab4866b376daf7e4871f658f65e7b79626711c0bd7bda7f0cb not found: ID does not exist" Sep 08 10:16:35.545994 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.545965 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx"] Sep 08 10:16:35.549771 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:35.549751 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-057291ac-openshift-default-59f858b896-q9jsx"] Sep 08 10:16:36.679707 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:36.679667 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" path="/var/lib/kubelet/pods/3250d43d-7f11-46bd-8d58-0c29ed51fc44/volumes" Sep 08 10:16:38.655015 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:38.654982 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:40.037109 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037075 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv"] Sep 08 10:16:40.037707 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037685 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerName="istio-proxy" Sep 08 10:16:40.037707 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037705 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerName="istio-proxy" Sep 08 10:16:40.037914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037723 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerName="istio-proxy" Sep 08 10:16:40.037914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037731 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerName="istio-proxy" Sep 08 10:16:40.037914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037754 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ba59c2-f394-4365-84e5-056739f115cc" containerName="istio-proxy" Sep 08 10:16:40.037914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037764 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ba59c2-f394-4365-84e5-056739f115cc" containerName="istio-proxy" Sep 08 10:16:40.037914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037774 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ba59c2-f394-4365-84e5-056739f115cc" containerName="istio-proxy" Sep 08 10:16:40.037914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037782 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ba59c2-f394-4365-84e5-056739f115cc" containerName="istio-proxy" Sep 08 10:16:40.037914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037799 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe67257e-980c-42f0-b627-dabd8532d2b2" containerName="maas-api" Sep 08 10:16:40.037914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037808 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe67257e-980c-42f0-b627-dabd8532d2b2" containerName="maas-api" Sep 08 10:16:40.037914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037890 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerName="istio-proxy" Sep 08 10:16:40.037914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037903 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3250d43d-7f11-46bd-8d58-0c29ed51fc44" containerName="istio-proxy" Sep 08 10:16:40.037914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037910 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe67257e-980c-42f0-b627-dabd8532d2b2" containerName="maas-api" Sep 08 10:16:40.037914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037919 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ba59c2-f394-4365-84e5-056739f115cc" containerName="istio-proxy" Sep 08 10:16:40.038301 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.037930 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ba59c2-f394-4365-84e5-056739f115cc" containerName="istio-proxy" Sep 08 10:16:40.042029 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.042011 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.045360 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.045342 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9e365b\"/\"default-dockercfg-pjghc\"" Sep 08 10:16:40.046238 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.046219 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9e365b\"/\"kube-root-ca.crt\"" Sep 08 10:16:40.046488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.046467 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9e365b\"/\"test-model-9e365b-kserve-self-signed-certs\"" Sep 08 10:16:40.046564 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.046498 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9e365b\"/\"openshift-service-ca.crt\"" Sep 08 10:16:40.054832 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.054813 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv"] Sep 08 10:16:40.134316 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.134283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f00334-0e57-4c85-bb97-a2ad90550e61-tls-certs\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.134492 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.134323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-tmp-dir\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.134492 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.134383 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-dshm\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.134492 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.134429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-home\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.134492 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.134467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-model-cache\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.134688 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.134495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p42c8\" (UniqueName: \"kubernetes.io/projected/08f00334-0e57-4c85-bb97-a2ad90550e61-kube-api-access-p42c8\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.235439 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.235403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f00334-0e57-4c85-bb97-a2ad90550e61-tls-certs\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.235439 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.235438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-tmp-dir\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.235719 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.235461 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-dshm\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.235719 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.235591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-home\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.235719 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.235676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-model-cache\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.235719 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.235703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p42c8\" (UniqueName: \"kubernetes.io/projected/08f00334-0e57-4c85-bb97-a2ad90550e61-kube-api-access-p42c8\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.235937 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.235896 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-tmp-dir\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.236002 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.235979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-home\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.236056 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.236016 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-model-cache\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.237861 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.237837 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-dshm\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.238090 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.238071 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f00334-0e57-4c85-bb97-a2ad90550e61-tls-certs\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.244287 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.244267 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p42c8\" (UniqueName: \"kubernetes.io/projected/08f00334-0e57-4c85-bb97-a2ad90550e61-kube-api-access-p42c8\") pod \"test-model-9e365b-kserve-7c48c7f597-wp9lv\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.351391 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.351306 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:40.649015 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:40.648989 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv"] Sep 08 10:16:40.650408 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:16:40.650384 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08f00334_0e57_4c85_bb97_a2ad90550e61.slice/crio-e2ff8cda562f47e16badeb79fdbd0f28b886234cc712354d7ec949e4cc79306f WatchSource:0}: Error finding container e2ff8cda562f47e16badeb79fdbd0f28b886234cc712354d7ec949e4cc79306f: Status 404 returned error can't find the container with id e2ff8cda562f47e16badeb79fdbd0f28b886234cc712354d7ec949e4cc79306f Sep 08 10:16:41.566399 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:41.566360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" event={"ID":"08f00334-0e57-4c85-bb97-a2ad90550e61","Type":"ContainerStarted","Data":"6d75fd89a9c7d7c51d352cd933dda28745daf5b5976fe1c055a6fa748542b3c1"} Sep 08 10:16:41.566399 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:41.566400 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" event={"ID":"08f00334-0e57-4c85-bb97-a2ad90550e61","Type":"ContainerStarted","Data":"e2ff8cda562f47e16badeb79fdbd0f28b886234cc712354d7ec949e4cc79306f"} Sep 08 10:16:41.587637 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:41.584544 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" podStartSLOduration=1.584525174 podStartE2EDuration="1.584525174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:16:41.581899708 +0000 UTC m=+911.370951507" watchObservedRunningTime="2026-09-08 10:16:41.584525174 +0000 UTC m=+911.373576979" Sep 08 10:16:44.769236 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:44.769200 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:16:45.053055 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:45.052965 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:16:50.352288 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:50.352252 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:50.352746 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:50.352303 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:50.364397 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:50.364368 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:16:50.633357 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:16:50.633273 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:17:04.046630 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:04.046586 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:17:05.364971 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:05.364920 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:05.465313 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:05.465277 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:11.206243 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:11.206210 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:11.306376 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:11.306343 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:11.399378 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:11.399343 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:21.155599 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:21.155558 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:17:27.339175 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:27.339143 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:33.658538 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:33.658504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:33.914145 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:33.914061 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:34.139666 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:34.139611 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:35.429329 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:35.429298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:39.393601 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:39.393554 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:17:44.541887 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:44.541852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:45.441062 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:45.441026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:47.820960 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:47.820919 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:51.219038 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:51.219003 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:52.807885 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:52.807847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:53.081836 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:53.081728 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:17:53.128767 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:53.128736 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:53.273468 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:53.273436 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:54.001898 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:54.001866 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:54.267432 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:54.267347 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:55.630272 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:55.630241 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:55.732416 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:55.732386 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:55.800562 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:55.800534 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:55.916970 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:55.916937 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:56.456245 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:56.456213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:56.900338 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:56.900310 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:57.043003 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:57.042961 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:58.634692 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:58.634658 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:58.766985 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:58.766948 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:17:58.908522 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:17:58.908496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:02.065242 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:02.065198 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:02.160767 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:02.160735 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:02.891955 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:02.891924 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:09.979394 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:09.979364 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:10.110211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:10.110162 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:10.265434 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:10.265349 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:10.742985 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:10.742897 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:10.906279 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:10.906247 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:13.148334 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:13.148299 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:14.413472 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:14.413430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:14.506775 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:14.506744 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:15.653012 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.652981 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp"] Sep 08 10:18:15.656798 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.656776 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:15.659844 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.659823 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0a3f5190-serving-cert\"" Sep 08 10:18:15.688068 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.688043 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp"] Sep 08 10:18:15.725445 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.725419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:15.832130 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.832096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb49aa3-5611-4408-944f-1290e1e51330-maas-api-tls\") pod \"maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:15.832294 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.832206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb49aa3-5611-4408-944f-1290e1e51330-openshift-service-ca\") pod \"maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:15.832294 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.832236 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb49aa3-5611-4408-944f-1290e1e51330-metrics-certs\") pod \"maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:15.832397 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.832374 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gwcs\" (UniqueName: \"kubernetes.io/projected/fcb49aa3-5611-4408-944f-1290e1e51330-kube-api-access-6gwcs\") pod \"maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:15.891304 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.891273 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:15.933389 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.933309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb49aa3-5611-4408-944f-1290e1e51330-maas-api-tls\") pod \"maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:15.933389 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.933363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb49aa3-5611-4408-944f-1290e1e51330-openshift-service-ca\") pod \"maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:15.933389 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.933381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb49aa3-5611-4408-944f-1290e1e51330-metrics-certs\") pod \"maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:15.933697 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.933417 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gwcs\" (UniqueName: \"kubernetes.io/projected/fcb49aa3-5611-4408-944f-1290e1e51330-kube-api-access-6gwcs\") pod \"maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:15.934064 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.934038 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb49aa3-5611-4408-944f-1290e1e51330-openshift-service-ca\") pod \"maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:15.935827 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.935804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb49aa3-5611-4408-944f-1290e1e51330-maas-api-tls\") pod \"maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:15.935941 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.935876 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb49aa3-5611-4408-944f-1290e1e51330-metrics-certs\") pod \"maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:15.945237 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.945214 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gwcs\" (UniqueName: \"kubernetes.io/projected/fcb49aa3-5611-4408-944f-1290e1e51330-kube-api-access-6gwcs\") pod \"maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:15.967577 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:15.967556 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:16.265084 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:16.265055 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp"] Sep 08 10:18:16.266592 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:18:16.266567 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb49aa3_5611_4408_944f_1290e1e51330.slice/crio-244341084dad45f308efb2c2ea5d1edfbe9f17f59087d03e2eacae452509ce2b WatchSource:0}: Error finding container 244341084dad45f308efb2c2ea5d1edfbe9f17f59087d03e2eacae452509ce2b: Status 404 returned error can't find the container with id 244341084dad45f308efb2c2ea5d1edfbe9f17f59087d03e2eacae452509ce2b Sep 08 10:18:16.777784 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:16.777751 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:16.885851 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:16.885814 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:17.131227 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:17.131122 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" event={"ID":"fcb49aa3-5611-4408-944f-1290e1e51330","Type":"ContainerStarted","Data":"194f0d89336008f8d9c7a497f25f81628011d52eb6ee452f994d64e91f4888f9"} Sep 08 10:18:17.131227 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:17.131187 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:17.131227 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:17.131212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" event={"ID":"fcb49aa3-5611-4408-944f-1290e1e51330","Type":"ContainerStarted","Data":"244341084dad45f308efb2c2ea5d1edfbe9f17f59087d03e2eacae452509ce2b"} Sep 08 10:18:21.573415 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:21.573378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:23.142751 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:23.142721 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:18:23.168005 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:23.167953 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" podStartSLOduration=7.642473665 podStartE2EDuration="8.167936957s" totalImagesPullingTime="525.463292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:18:17.205223185 +0000 UTC m=+1006.994274990" watchObservedRunningTime="2026-09-08 10:18:23.167936957 +0000 UTC m=+1012.956988807" Sep 08 10:18:25.179100 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:25.179067 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:18:31.430015 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:31.429970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:32.389327 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:32.389289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:32.501427 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:32.501397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:32.755156 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:32.755128 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:32.847632 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:32.847580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:33.452075 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:33.452045 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv"] Sep 08 10:18:36.678676 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:36.678645 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:36.749550 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:36.749517 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:37.916551 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:37.916521 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:39.073518 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:39.073487 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:42.797833 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:42.797793 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:18:43.969178 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:43.969136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:44.563588 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:44.563554 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:44.723805 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:44.723772 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh"] Sep 08 10:18:44.724049 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:44.724023 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" podUID="6ad79fe7-9c86-4583-88f0-86ec392540e7" containerName="maas-api" containerID="cri-o://83b2d16ab06f7b086a0234731dee91297999e56c2856415316940789ea0cab96" gracePeriod=30 Sep 08 10:18:44.819553 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:44.819469 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:44.936357 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:44.936326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:45.117877 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.117837 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:45.178412 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.178390 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:18:45.293385 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.293355 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:45.298005 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.297978 2576 generic.go:356] "Generic (PLEG): container finished" podID="6ad79fe7-9c86-4583-88f0-86ec392540e7" containerID="83b2d16ab06f7b086a0234731dee91297999e56c2856415316940789ea0cab96" exitCode=0 Sep 08 10:18:45.298132 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.298034 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" Sep 08 10:18:45.298132 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.298055 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" event={"ID":"6ad79fe7-9c86-4583-88f0-86ec392540e7","Type":"ContainerDied","Data":"83b2d16ab06f7b086a0234731dee91297999e56c2856415316940789ea0cab96"} Sep 08 10:18:45.298132 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.298107 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh" event={"ID":"6ad79fe7-9c86-4583-88f0-86ec392540e7","Type":"ContainerDied","Data":"3441778e912d9868d88a4bc1b7000e3e23dad87969b501f7afaa7ac177588a9b"} Sep 08 10:18:45.298132 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.298131 2576 scope.go:117] "RemoveContainer" containerID="83b2d16ab06f7b086a0234731dee91297999e56c2856415316940789ea0cab96" Sep 08 10:18:45.307471 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.307453 2576 scope.go:117] "RemoveContainer" containerID="83b2d16ab06f7b086a0234731dee91297999e56c2856415316940789ea0cab96" Sep 08 10:18:45.307785 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:18:45.307758 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b2d16ab06f7b086a0234731dee91297999e56c2856415316940789ea0cab96\": container with ID starting with 83b2d16ab06f7b086a0234731dee91297999e56c2856415316940789ea0cab96 not found: ID does not exist" containerID="83b2d16ab06f7b086a0234731dee91297999e56c2856415316940789ea0cab96" Sep 08 10:18:45.307836 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.307796 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b2d16ab06f7b086a0234731dee91297999e56c2856415316940789ea0cab96"} err="failed to get container status \"83b2d16ab06f7b086a0234731dee91297999e56c2856415316940789ea0cab96\": rpc error: code = NotFound desc = could not find container \"83b2d16ab06f7b086a0234731dee91297999e56c2856415316940789ea0cab96\": container with ID starting with 83b2d16ab06f7b086a0234731dee91297999e56c2856415316940789ea0cab96 not found: ID does not exist" Sep 08 10:18:45.311085 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.311068 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6wz2\" (UniqueName: \"kubernetes.io/projected/6ad79fe7-9c86-4583-88f0-86ec392540e7-kube-api-access-v6wz2\") pod \"6ad79fe7-9c86-4583-88f0-86ec392540e7\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " Sep 08 10:18:45.311186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.311173 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad79fe7-9c86-4583-88f0-86ec392540e7-openshift-service-ca\") pod \"6ad79fe7-9c86-4583-88f0-86ec392540e7\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " Sep 08 10:18:45.311225 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.311209 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ad79fe7-9c86-4583-88f0-86ec392540e7-metrics-certs\") pod \"6ad79fe7-9c86-4583-88f0-86ec392540e7\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " Sep 08 10:18:45.311266 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.311253 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad79fe7-9c86-4583-88f0-86ec392540e7-maas-api-tls\") pod \"6ad79fe7-9c86-4583-88f0-86ec392540e7\" (UID: \"6ad79fe7-9c86-4583-88f0-86ec392540e7\") " Sep 08 10:18:45.311552 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.311525 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad79fe7-9c86-4583-88f0-86ec392540e7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6ad79fe7-9c86-4583-88f0-86ec392540e7" (UID: "6ad79fe7-9c86-4583-88f0-86ec392540e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:18:45.313072 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.313044 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad79fe7-9c86-4583-88f0-86ec392540e7-kube-api-access-v6wz2" (OuterVolumeSpecName: "kube-api-access-v6wz2") pod "6ad79fe7-9c86-4583-88f0-86ec392540e7" (UID: "6ad79fe7-9c86-4583-88f0-86ec392540e7"). InnerVolumeSpecName "kube-api-access-v6wz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:18:45.313163 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.313146 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad79fe7-9c86-4583-88f0-86ec392540e7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6ad79fe7-9c86-4583-88f0-86ec392540e7" (UID: "6ad79fe7-9c86-4583-88f0-86ec392540e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:18:45.313325 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.313311 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad79fe7-9c86-4583-88f0-86ec392540e7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6ad79fe7-9c86-4583-88f0-86ec392540e7" (UID: "6ad79fe7-9c86-4583-88f0-86ec392540e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:18:45.412594 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.412561 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad79fe7-9c86-4583-88f0-86ec392540e7-openshift-service-ca\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:45.412594 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.412595 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ad79fe7-9c86-4583-88f0-86ec392540e7-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:45.412813 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.412604 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad79fe7-9c86-4583-88f0-86ec392540e7-maas-api-tls\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:45.412813 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.412611 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6wz2\" (UniqueName: \"kubernetes.io/projected/6ad79fe7-9c86-4583-88f0-86ec392540e7-kube-api-access-v6wz2\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:45.416789 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.416762 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:45.636764 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.636729 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:45.638302 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.638281 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh"] Sep 08 10:18:45.652383 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.652349 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c28013-7f4b64d585-hxnrh"] Sep 08 10:18:45.840824 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:45.840741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:46.678727 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:46.678696 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ad79fe7-9c86-4583-88f0-86ec392540e7" path="/var/lib/kubelet/pods/6ad79fe7-9c86-4583-88f0-86ec392540e7/volumes" Sep 08 10:18:48.471720 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.471679 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" podUID="08f00334-0e57-4c85-bb97-a2ad90550e61" containerName="main" containerID="cri-o://6d75fd89a9c7d7c51d352cd933dda28745daf5b5976fe1c055a6fa748542b3c1" gracePeriod=45 Sep 08 10:18:48.726652 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.726583 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:18:48.845029 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.844991 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-home\") pod \"08f00334-0e57-4c85-bb97-a2ad90550e61\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " Sep 08 10:18:48.845216 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.845060 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-model-cache\") pod \"08f00334-0e57-4c85-bb97-a2ad90550e61\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " Sep 08 10:18:48.845216 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.845100 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-tmp-dir\") pod \"08f00334-0e57-4c85-bb97-a2ad90550e61\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " Sep 08 10:18:48.845216 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.845157 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f00334-0e57-4c85-bb97-a2ad90550e61-tls-certs\") pod \"08f00334-0e57-4c85-bb97-a2ad90550e61\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " Sep 08 10:18:48.845216 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.845181 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p42c8\" (UniqueName: \"kubernetes.io/projected/08f00334-0e57-4c85-bb97-a2ad90550e61-kube-api-access-p42c8\") pod \"08f00334-0e57-4c85-bb97-a2ad90550e61\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " Sep 08 10:18:48.845404 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.845306 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-home" (OuterVolumeSpecName: "home") pod "08f00334-0e57-4c85-bb97-a2ad90550e61" (UID: "08f00334-0e57-4c85-bb97-a2ad90550e61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:18:48.845404 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.845325 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-model-cache" (OuterVolumeSpecName: "model-cache") pod "08f00334-0e57-4c85-bb97-a2ad90550e61" (UID: "08f00334-0e57-4c85-bb97-a2ad90550e61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:18:48.845404 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.845365 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-dshm\") pod \"08f00334-0e57-4c85-bb97-a2ad90550e61\" (UID: \"08f00334-0e57-4c85-bb97-a2ad90550e61\") " Sep 08 10:18:48.845560 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.845405 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08f00334-0e57-4c85-bb97-a2ad90550e61" (UID: "08f00334-0e57-4c85-bb97-a2ad90550e61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:18:48.845726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.845703 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-home\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:48.845853 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.845727 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-model-cache\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:48.845853 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.845743 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-tmp-dir\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:48.847469 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.847445 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f00334-0e57-4c85-bb97-a2ad90550e61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08f00334-0e57-4c85-bb97-a2ad90550e61" (UID: "08f00334-0e57-4c85-bb97-a2ad90550e61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:18:48.847571 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.847463 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f00334-0e57-4c85-bb97-a2ad90550e61-kube-api-access-p42c8" (OuterVolumeSpecName: "kube-api-access-p42c8") pod "08f00334-0e57-4c85-bb97-a2ad90550e61" (UID: "08f00334-0e57-4c85-bb97-a2ad90550e61"). InnerVolumeSpecName "kube-api-access-p42c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:18:48.847571 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.847472 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-dshm" (OuterVolumeSpecName: "dshm") pod "08f00334-0e57-4c85-bb97-a2ad90550e61" (UID: "08f00334-0e57-4c85-bb97-a2ad90550e61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:18:48.946882 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.946850 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f00334-0e57-4c85-bb97-a2ad90550e61-tls-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:48.946882 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.946879 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p42c8\" (UniqueName: \"kubernetes.io/projected/08f00334-0e57-4c85-bb97-a2ad90550e61-kube-api-access-p42c8\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:48.947063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:48.946890 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f00334-0e57-4c85-bb97-a2ad90550e61-dshm\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:49.322767 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:49.322738 2576 generic.go:356] "Generic (PLEG): container finished" podID="08f00334-0e57-4c85-bb97-a2ad90550e61" containerID="6d75fd89a9c7d7c51d352cd933dda28745daf5b5976fe1c055a6fa748542b3c1" exitCode=0 Sep 08 10:18:49.322946 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:49.322775 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" Sep 08 10:18:49.322946 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:49.322820 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" event={"ID":"08f00334-0e57-4c85-bb97-a2ad90550e61","Type":"ContainerDied","Data":"6d75fd89a9c7d7c51d352cd933dda28745daf5b5976fe1c055a6fa748542b3c1"} Sep 08 10:18:49.322946 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:49.322861 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv" event={"ID":"08f00334-0e57-4c85-bb97-a2ad90550e61","Type":"ContainerDied","Data":"e2ff8cda562f47e16badeb79fdbd0f28b886234cc712354d7ec949e4cc79306f"} Sep 08 10:18:49.322946 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:49.322877 2576 scope.go:117] "RemoveContainer" containerID="6d75fd89a9c7d7c51d352cd933dda28745daf5b5976fe1c055a6fa748542b3c1" Sep 08 10:18:49.332476 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:49.332460 2576 scope.go:117] "RemoveContainer" containerID="6d75fd89a9c7d7c51d352cd933dda28745daf5b5976fe1c055a6fa748542b3c1" Sep 08 10:18:49.332751 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:18:49.332733 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d75fd89a9c7d7c51d352cd933dda28745daf5b5976fe1c055a6fa748542b3c1\": container with ID starting with 6d75fd89a9c7d7c51d352cd933dda28745daf5b5976fe1c055a6fa748542b3c1 not found: ID does not exist" containerID="6d75fd89a9c7d7c51d352cd933dda28745daf5b5976fe1c055a6fa748542b3c1" Sep 08 10:18:49.332797 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:49.332759 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d75fd89a9c7d7c51d352cd933dda28745daf5b5976fe1c055a6fa748542b3c1"} err="failed to get container status \"6d75fd89a9c7d7c51d352cd933dda28745daf5b5976fe1c055a6fa748542b3c1\": rpc error: code = NotFound desc = could not find container \"6d75fd89a9c7d7c51d352cd933dda28745daf5b5976fe1c055a6fa748542b3c1\": container with ID starting with 6d75fd89a9c7d7c51d352cd933dda28745daf5b5976fe1c055a6fa748542b3c1 not found: ID does not exist" Sep 08 10:18:49.344673 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:49.344653 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv"] Sep 08 10:18:49.348054 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:49.348033 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9e365b/test-model-9e365b-kserve-7c48c7f597-wp9lv"] Sep 08 10:18:50.678657 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:50.678608 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08f00334-0e57-4c85-bb97-a2ad90550e61" path="/var/lib/kubelet/pods/08f00334-0e57-4c85-bb97-a2ad90550e61/volumes" Sep 08 10:18:52.306688 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:52.306638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:52.403504 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:52.403470 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:52.505440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:52.505412 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:52.562565 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:52.562487 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc"] Sep 08 10:18:52.562771 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:52.562749 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" podUID="cc97ae01-9034-497d-9b1b-e34c8997e485" containerName="istio-proxy" containerID="cri-o://b5f0fe2010924b8a26c2433177760bb8003ec5927e73469ce9d3fa522960a5db" gracePeriod=30 Sep 08 10:18:52.635947 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:52.635912 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:52.939346 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:52.939313 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:54.097434 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:54.097394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:55.546030 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:55.545995 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:18:57.832094 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.832037 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:18:57.933463 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.933433 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpnr2\" (UniqueName: \"kubernetes.io/projected/cc97ae01-9034-497d-9b1b-e34c8997e485-kube-api-access-jpnr2\") pod \"cc97ae01-9034-497d-9b1b-e34c8997e485\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " Sep 08 10:18:57.933463 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.933468 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-credential-socket\") pod \"cc97ae01-9034-497d-9b1b-e34c8997e485\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " Sep 08 10:18:57.933741 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.933516 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-workload-socket\") pod \"cc97ae01-9034-497d-9b1b-e34c8997e485\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " Sep 08 10:18:57.933741 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.933537 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc97ae01-9034-497d-9b1b-e34c8997e485-istiod-ca-cert\") pod \"cc97ae01-9034-497d-9b1b-e34c8997e485\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " Sep 08 10:18:57.933741 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.933588 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-envoy\") pod \"cc97ae01-9034-497d-9b1b-e34c8997e485\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " Sep 08 10:18:57.933741 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.933644 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-podinfo\") pod \"cc97ae01-9034-497d-9b1b-e34c8997e485\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " Sep 08 10:18:57.933741 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.933680 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-data\") pod \"cc97ae01-9034-497d-9b1b-e34c8997e485\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " Sep 08 10:18:57.933741 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.933716 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-token\") pod \"cc97ae01-9034-497d-9b1b-e34c8997e485\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " Sep 08 10:18:57.933741 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.933732 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-workload-certs\") pod \"cc97ae01-9034-497d-9b1b-e34c8997e485\" (UID: \"cc97ae01-9034-497d-9b1b-e34c8997e485\") " Sep 08 10:18:57.934317 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.933887 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cc97ae01-9034-497d-9b1b-e34c8997e485" (UID: "cc97ae01-9034-497d-9b1b-e34c8997e485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:18:57.934317 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.933996 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc97ae01-9034-497d-9b1b-e34c8997e485-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cc97ae01-9034-497d-9b1b-e34c8997e485" (UID: "cc97ae01-9034-497d-9b1b-e34c8997e485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:18:57.934317 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.933877 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cc97ae01-9034-497d-9b1b-e34c8997e485" (UID: "cc97ae01-9034-497d-9b1b-e34c8997e485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:18:57.934317 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.934129 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cc97ae01-9034-497d-9b1b-e34c8997e485" (UID: "cc97ae01-9034-497d-9b1b-e34c8997e485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:18:57.934605 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.934582 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-data" (OuterVolumeSpecName: "istio-data") pod "cc97ae01-9034-497d-9b1b-e34c8997e485" (UID: "cc97ae01-9034-497d-9b1b-e34c8997e485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:18:57.936013 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.935988 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-workload-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:57.936168 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.936015 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-credential-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:57.936168 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.936030 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-workload-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:57.936168 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.936043 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc97ae01-9034-497d-9b1b-e34c8997e485-istiod-ca-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:57.936168 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.936056 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-data\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:57.936168 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.936105 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cc97ae01-9034-497d-9b1b-e34c8997e485" (UID: "cc97ae01-9034-497d-9b1b-e34c8997e485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:18:57.936568 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.936546 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc97ae01-9034-497d-9b1b-e34c8997e485-kube-api-access-jpnr2" (OuterVolumeSpecName: "kube-api-access-jpnr2") pod "cc97ae01-9034-497d-9b1b-e34c8997e485" (UID: "cc97ae01-9034-497d-9b1b-e34c8997e485"). InnerVolumeSpecName "kube-api-access-jpnr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:18:57.936568 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.936552 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cc97ae01-9034-497d-9b1b-e34c8997e485" (UID: "cc97ae01-9034-497d-9b1b-e34c8997e485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 10:18:57.936748 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:57.936574 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-token" (OuterVolumeSpecName: "istio-token") pod "cc97ae01-9034-497d-9b1b-e34c8997e485" (UID: "cc97ae01-9034-497d-9b1b-e34c8997e485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:18:58.036556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.036521 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-envoy\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:58.036556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.036550 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-podinfo\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:58.036556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.036561 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc97ae01-9034-497d-9b1b-e34c8997e485-istio-token\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:58.036807 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.036574 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpnr2\" (UniqueName: \"kubernetes.io/projected/cc97ae01-9034-497d-9b1b-e34c8997e485-kube-api-access-jpnr2\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:18:58.143210 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.143176 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:18:58.377112 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.377081 2576 generic.go:356] "Generic (PLEG): container finished" podID="cc97ae01-9034-497d-9b1b-e34c8997e485" containerID="b5f0fe2010924b8a26c2433177760bb8003ec5927e73469ce9d3fa522960a5db" exitCode=0 Sep 08 10:18:58.377289 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.377128 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" Sep 08 10:18:58.377289 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.377163 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" event={"ID":"cc97ae01-9034-497d-9b1b-e34c8997e485","Type":"ContainerDied","Data":"b5f0fe2010924b8a26c2433177760bb8003ec5927e73469ce9d3fa522960a5db"} Sep 08 10:18:58.377289 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.377218 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc" event={"ID":"cc97ae01-9034-497d-9b1b-e34c8997e485","Type":"ContainerDied","Data":"9c4e3f8c4d5df6b3099886af419fc72710a21f6f6b40dbc1d9ecc88e7d6d98d3"} Sep 08 10:18:58.377289 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.377240 2576 scope.go:117] "RemoveContainer" containerID="b5f0fe2010924b8a26c2433177760bb8003ec5927e73469ce9d3fa522960a5db" Sep 08 10:18:58.394754 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.394733 2576 scope.go:117] "RemoveContainer" containerID="e3a9dfb7938225fa18aa151de3c195b1b4c31e3deafbef23141df43685f5c103" Sep 08 10:18:58.402180 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.402155 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc"] Sep 08 10:18:58.404009 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.403990 2576 scope.go:117] "RemoveContainer" containerID="b5f0fe2010924b8a26c2433177760bb8003ec5927e73469ce9d3fa522960a5db" Sep 08 10:18:58.404347 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:18:58.404324 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f0fe2010924b8a26c2433177760bb8003ec5927e73469ce9d3fa522960a5db\": container with ID starting with b5f0fe2010924b8a26c2433177760bb8003ec5927e73469ce9d3fa522960a5db not found: ID does not exist" containerID="b5f0fe2010924b8a26c2433177760bb8003ec5927e73469ce9d3fa522960a5db" Sep 08 10:18:58.404408 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.404359 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f0fe2010924b8a26c2433177760bb8003ec5927e73469ce9d3fa522960a5db"} err="failed to get container status \"b5f0fe2010924b8a26c2433177760bb8003ec5927e73469ce9d3fa522960a5db\": rpc error: code = NotFound desc = could not find container \"b5f0fe2010924b8a26c2433177760bb8003ec5927e73469ce9d3fa522960a5db\": container with ID starting with b5f0fe2010924b8a26c2433177760bb8003ec5927e73469ce9d3fa522960a5db not found: ID does not exist" Sep 08 10:18:58.404408 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.404384 2576 scope.go:117] "RemoveContainer" containerID="e3a9dfb7938225fa18aa151de3c195b1b4c31e3deafbef23141df43685f5c103" Sep 08 10:18:58.404682 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:18:58.404661 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a9dfb7938225fa18aa151de3c195b1b4c31e3deafbef23141df43685f5c103\": container with ID starting with e3a9dfb7938225fa18aa151de3c195b1b4c31e3deafbef23141df43685f5c103 not found: ID does not exist" containerID="e3a9dfb7938225fa18aa151de3c195b1b4c31e3deafbef23141df43685f5c103" Sep 08 10:18:58.404751 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.404691 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a9dfb7938225fa18aa151de3c195b1b4c31e3deafbef23141df43685f5c103"} err="failed to get container status \"e3a9dfb7938225fa18aa151de3c195b1b4c31e3deafbef23141df43685f5c103\": rpc error: code = NotFound desc = could not find container \"e3a9dfb7938225fa18aa151de3c195b1b4c31e3deafbef23141df43685f5c103\": container with ID starting with e3a9dfb7938225fa18aa151de3c195b1b4c31e3deafbef23141df43685f5c103 not found: ID does not exist" Sep 08 10:18:58.405195 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.405177 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c28013-openshift-default-fb8c779d4-9f4sc"] Sep 08 10:18:58.678346 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:18:58.678264 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc97ae01-9034-497d-9b1b-e34c8997e485" path="/var/lib/kubelet/pods/cc97ae01-9034-497d-9b1b-e34c8997e485/volumes" Sep 08 10:19:03.426945 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.426915 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr"] Sep 08 10:19:03.427338 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.427107 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" podUID="0526bd46-f2e0-4c3c-8386-2043f72eab6d" containerName="maas-api" containerID="cri-o://d80955a1c9b583620c485e7e9d85391c1767aa445ce2f12cd2edfa3339c23c52" gracePeriod=30 Sep 08 10:19:03.586217 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.586177 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp"] Sep 08 10:19:03.586778 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.586465 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" podUID="fcb49aa3-5611-4408-944f-1290e1e51330" containerName="maas-api" containerID="cri-o://194f0d89336008f8d9c7a497f25f81628011d52eb6ee452f994d64e91f4888f9" gracePeriod=30 Sep 08 10:19:03.739019 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.738986 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:19:03.788288 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.788261 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdvmc\" (UniqueName: \"kubernetes.io/projected/0526bd46-f2e0-4c3c-8386-2043f72eab6d-kube-api-access-mdvmc\") pod \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " Sep 08 10:19:03.788426 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.788344 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0526bd46-f2e0-4c3c-8386-2043f72eab6d-openshift-service-ca\") pod \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " Sep 08 10:19:03.788426 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.788375 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0526bd46-f2e0-4c3c-8386-2043f72eab6d-maas-api-tls\") pod \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " Sep 08 10:19:03.788543 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.788428 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0526bd46-f2e0-4c3c-8386-2043f72eab6d-metrics-certs\") pod \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\" (UID: \"0526bd46-f2e0-4c3c-8386-2043f72eab6d\") " Sep 08 10:19:03.788786 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.788756 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0526bd46-f2e0-4c3c-8386-2043f72eab6d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0526bd46-f2e0-4c3c-8386-2043f72eab6d" (UID: "0526bd46-f2e0-4c3c-8386-2043f72eab6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:19:03.793781 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.793748 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0526bd46-f2e0-4c3c-8386-2043f72eab6d-kube-api-access-mdvmc" (OuterVolumeSpecName: "kube-api-access-mdvmc") pod "0526bd46-f2e0-4c3c-8386-2043f72eab6d" (UID: "0526bd46-f2e0-4c3c-8386-2043f72eab6d"). InnerVolumeSpecName "kube-api-access-mdvmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:19:03.794820 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.794790 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0526bd46-f2e0-4c3c-8386-2043f72eab6d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0526bd46-f2e0-4c3c-8386-2043f72eab6d" (UID: "0526bd46-f2e0-4c3c-8386-2043f72eab6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:19:03.794942 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.794895 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0526bd46-f2e0-4c3c-8386-2043f72eab6d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0526bd46-f2e0-4c3c-8386-2043f72eab6d" (UID: "0526bd46-f2e0-4c3c-8386-2043f72eab6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:19:03.854387 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.854364 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:19:03.889481 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.889453 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb49aa3-5611-4408-944f-1290e1e51330-metrics-certs\") pod \"fcb49aa3-5611-4408-944f-1290e1e51330\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " Sep 08 10:19:03.889481 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.889499 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb49aa3-5611-4408-944f-1290e1e51330-openshift-service-ca\") pod \"fcb49aa3-5611-4408-944f-1290e1e51330\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " Sep 08 10:19:03.889831 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.889651 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb49aa3-5611-4408-944f-1290e1e51330-maas-api-tls\") pod \"fcb49aa3-5611-4408-944f-1290e1e51330\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " Sep 08 10:19:03.889831 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.889686 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gwcs\" (UniqueName: \"kubernetes.io/projected/fcb49aa3-5611-4408-944f-1290e1e51330-kube-api-access-6gwcs\") pod \"fcb49aa3-5611-4408-944f-1290e1e51330\" (UID: \"fcb49aa3-5611-4408-944f-1290e1e51330\") " Sep 08 10:19:03.890006 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.889983 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0526bd46-f2e0-4c3c-8386-2043f72eab6d-openshift-service-ca\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:19:03.890183 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.890010 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0526bd46-f2e0-4c3c-8386-2043f72eab6d-maas-api-tls\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:19:03.890183 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.890025 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0526bd46-f2e0-4c3c-8386-2043f72eab6d-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:19:03.890183 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.890038 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdvmc\" (UniqueName: \"kubernetes.io/projected/0526bd46-f2e0-4c3c-8386-2043f72eab6d-kube-api-access-mdvmc\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:19:03.890360 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.890336 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb49aa3-5611-4408-944f-1290e1e51330-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fcb49aa3-5611-4408-944f-1290e1e51330" (UID: "fcb49aa3-5611-4408-944f-1290e1e51330"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:19:03.892311 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.892283 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb49aa3-5611-4408-944f-1290e1e51330-kube-api-access-6gwcs" (OuterVolumeSpecName: "kube-api-access-6gwcs") pod "fcb49aa3-5611-4408-944f-1290e1e51330" (UID: "fcb49aa3-5611-4408-944f-1290e1e51330"). InnerVolumeSpecName "kube-api-access-6gwcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:19:03.892802 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.892779 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb49aa3-5611-4408-944f-1290e1e51330-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fcb49aa3-5611-4408-944f-1290e1e51330" (UID: "fcb49aa3-5611-4408-944f-1290e1e51330"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:19:03.892869 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.892788 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb49aa3-5611-4408-944f-1290e1e51330-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fcb49aa3-5611-4408-944f-1290e1e51330" (UID: "fcb49aa3-5611-4408-944f-1290e1e51330"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:19:03.991585 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.991492 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb49aa3-5611-4408-944f-1290e1e51330-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:19:03.991585 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.991530 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb49aa3-5611-4408-944f-1290e1e51330-openshift-service-ca\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:19:03.991585 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.991547 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb49aa3-5611-4408-944f-1290e1e51330-maas-api-tls\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:19:03.991585 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:03.991562 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gwcs\" (UniqueName: \"kubernetes.io/projected/fcb49aa3-5611-4408-944f-1290e1e51330-kube-api-access-6gwcs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:19:04.416493 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.416456 2576 generic.go:356] "Generic (PLEG): container finished" podID="fcb49aa3-5611-4408-944f-1290e1e51330" containerID="194f0d89336008f8d9c7a497f25f81628011d52eb6ee452f994d64e91f4888f9" exitCode=0 Sep 08 10:19:04.416651 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.416502 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" Sep 08 10:19:04.416651 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.416549 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" event={"ID":"fcb49aa3-5611-4408-944f-1290e1e51330","Type":"ContainerDied","Data":"194f0d89336008f8d9c7a497f25f81628011d52eb6ee452f994d64e91f4888f9"} Sep 08 10:19:04.416651 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.416646 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp" event={"ID":"fcb49aa3-5611-4408-944f-1290e1e51330","Type":"ContainerDied","Data":"244341084dad45f308efb2c2ea5d1edfbe9f17f59087d03e2eacae452509ce2b"} Sep 08 10:19:04.416802 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.416675 2576 scope.go:117] "RemoveContainer" containerID="194f0d89336008f8d9c7a497f25f81628011d52eb6ee452f994d64e91f4888f9" Sep 08 10:19:04.417917 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.417892 2576 generic.go:356] "Generic (PLEG): container finished" podID="0526bd46-f2e0-4c3c-8386-2043f72eab6d" containerID="d80955a1c9b583620c485e7e9d85391c1767aa445ce2f12cd2edfa3339c23c52" exitCode=0 Sep 08 10:19:04.417917 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.417924 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" Sep 08 10:19:04.418081 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.417940 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" event={"ID":"0526bd46-f2e0-4c3c-8386-2043f72eab6d","Type":"ContainerDied","Data":"d80955a1c9b583620c485e7e9d85391c1767aa445ce2f12cd2edfa3339c23c52"} Sep 08 10:19:04.418081 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.417983 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr" event={"ID":"0526bd46-f2e0-4c3c-8386-2043f72eab6d","Type":"ContainerDied","Data":"58bc7347a1ad113c2b0fcb322186597f95de1648a36ce6786c29e92a1fdb14c5"} Sep 08 10:19:04.429997 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.429795 2576 scope.go:117] "RemoveContainer" containerID="194f0d89336008f8d9c7a497f25f81628011d52eb6ee452f994d64e91f4888f9" Sep 08 10:19:04.430247 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:19:04.430051 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"194f0d89336008f8d9c7a497f25f81628011d52eb6ee452f994d64e91f4888f9\": container with ID starting with 194f0d89336008f8d9c7a497f25f81628011d52eb6ee452f994d64e91f4888f9 not found: ID does not exist" containerID="194f0d89336008f8d9c7a497f25f81628011d52eb6ee452f994d64e91f4888f9" Sep 08 10:19:04.430247 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.430072 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"194f0d89336008f8d9c7a497f25f81628011d52eb6ee452f994d64e91f4888f9"} err="failed to get container status \"194f0d89336008f8d9c7a497f25f81628011d52eb6ee452f994d64e91f4888f9\": rpc error: code = NotFound desc = could not find container \"194f0d89336008f8d9c7a497f25f81628011d52eb6ee452f994d64e91f4888f9\": container with ID starting with 194f0d89336008f8d9c7a497f25f81628011d52eb6ee452f994d64e91f4888f9 not found: ID does not exist" Sep 08 10:19:04.430247 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.430087 2576 scope.go:117] "RemoveContainer" containerID="d80955a1c9b583620c485e7e9d85391c1767aa445ce2f12cd2edfa3339c23c52" Sep 08 10:19:04.438462 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.438442 2576 scope.go:117] "RemoveContainer" containerID="d80955a1c9b583620c485e7e9d85391c1767aa445ce2f12cd2edfa3339c23c52" Sep 08 10:19:04.438729 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:19:04.438712 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80955a1c9b583620c485e7e9d85391c1767aa445ce2f12cd2edfa3339c23c52\": container with ID starting with d80955a1c9b583620c485e7e9d85391c1767aa445ce2f12cd2edfa3339c23c52 not found: ID does not exist" containerID="d80955a1c9b583620c485e7e9d85391c1767aa445ce2f12cd2edfa3339c23c52" Sep 08 10:19:04.438779 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.438733 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80955a1c9b583620c485e7e9d85391c1767aa445ce2f12cd2edfa3339c23c52"} err="failed to get container status \"d80955a1c9b583620c485e7e9d85391c1767aa445ce2f12cd2edfa3339c23c52\": rpc error: code = NotFound desc = could not find container \"d80955a1c9b583620c485e7e9d85391c1767aa445ce2f12cd2edfa3339c23c52\": container with ID starting with d80955a1c9b583620c485e7e9d85391c1767aa445ce2f12cd2edfa3339c23c52 not found: ID does not exist" Sep 08 10:19:04.443233 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.443213 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp"] Sep 08 10:19:04.449700 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.449677 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a3f5190-68b6bc9fbf-ktzhp"] Sep 08 10:19:04.460272 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.460252 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr"] Sep 08 10:19:04.465083 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.465064 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e365b-86c4ddd45f-v42hr"] Sep 08 10:19:04.678014 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.677919 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0526bd46-f2e0-4c3c-8386-2043f72eab6d" path="/var/lib/kubelet/pods/0526bd46-f2e0-4c3c-8386-2043f72eab6d/volumes" Sep 08 10:19:04.678420 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:04.678400 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcb49aa3-5611-4408-944f-1290e1e51330" path="/var/lib/kubelet/pods/fcb49aa3-5611-4408-944f-1290e1e51330/volumes" Sep 08 10:19:05.952271 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:05.952230 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:19:06.825388 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:06.825358 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:06.893491 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:06.893459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:11.534982 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.534948 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh"] Sep 08 10:19:11.535435 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535400 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc97ae01-9034-497d-9b1b-e34c8997e485" containerName="istio-proxy" Sep 08 10:19:11.535435 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535415 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc97ae01-9034-497d-9b1b-e34c8997e485" containerName="istio-proxy" Sep 08 10:19:11.535435 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535423 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08f00334-0e57-4c85-bb97-a2ad90550e61" containerName="main" Sep 08 10:19:11.535435 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535429 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="08f00334-0e57-4c85-bb97-a2ad90550e61" containerName="main" Sep 08 10:19:11.535435 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535437 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0526bd46-f2e0-4c3c-8386-2043f72eab6d" containerName="maas-api" Sep 08 10:19:11.535645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535442 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0526bd46-f2e0-4c3c-8386-2043f72eab6d" containerName="maas-api" Sep 08 10:19:11.535645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535449 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcb49aa3-5611-4408-944f-1290e1e51330" containerName="maas-api" Sep 08 10:19:11.535645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535454 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcb49aa3-5611-4408-944f-1290e1e51330" containerName="maas-api" Sep 08 10:19:11.535645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535462 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc97ae01-9034-497d-9b1b-e34c8997e485" containerName="istio-proxy" Sep 08 10:19:11.535645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535467 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc97ae01-9034-497d-9b1b-e34c8997e485" containerName="istio-proxy" Sep 08 10:19:11.535645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535484 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad79fe7-9c86-4583-88f0-86ec392540e7" containerName="maas-api" Sep 08 10:19:11.535645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535489 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad79fe7-9c86-4583-88f0-86ec392540e7" containerName="maas-api" Sep 08 10:19:11.535645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535563 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc97ae01-9034-497d-9b1b-e34c8997e485" containerName="istio-proxy" Sep 08 10:19:11.535645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535570 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0526bd46-f2e0-4c3c-8386-2043f72eab6d" containerName="maas-api" Sep 08 10:19:11.535645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535575 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad79fe7-9c86-4583-88f0-86ec392540e7" containerName="maas-api" Sep 08 10:19:11.535645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535582 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcb49aa3-5611-4408-944f-1290e1e51330" containerName="maas-api" Sep 08 10:19:11.535645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.535588 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="08f00334-0e57-4c85-bb97-a2ad90550e61" containerName="main" Sep 08 10:19:11.540303 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.540279 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.543139 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.543119 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-71a7a1-openshift-default-dockercfg-72mzb\"" Sep 08 10:19:11.553603 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.553423 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh"] Sep 08 10:19:11.558094 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.558061 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-credential-socket\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.558216 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.558198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krksw\" (UniqueName: \"kubernetes.io/projected/26271269-74fa-4615-bb71-085a510fe223-kube-api-access-krksw\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.558281 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.558256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26271269-74fa-4615-bb71-085a510fe223-istio-podinfo\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.558333 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.558288 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-workload-socket\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.558385 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.558330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-istio-envoy\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.558385 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.558359 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26271269-74fa-4615-bb71-085a510fe223-istio-token\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.558467 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.558383 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26271269-74fa-4615-bb71-085a510fe223-istiod-ca-cert\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.558501 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.558461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-workload-certs\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.558550 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.558512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-istio-data\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.659198 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.659162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krksw\" (UniqueName: \"kubernetes.io/projected/26271269-74fa-4615-bb71-085a510fe223-kube-api-access-krksw\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.659374 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.659209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26271269-74fa-4615-bb71-085a510fe223-istio-podinfo\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.659374 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.659232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-workload-socket\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.659374 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.659256 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-istio-envoy\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.659374 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.659274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26271269-74fa-4615-bb71-085a510fe223-istio-token\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.659374 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.659299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26271269-74fa-4615-bb71-085a510fe223-istiod-ca-cert\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.659374 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.659340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-workload-certs\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.659374 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.659377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-istio-data\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.659742 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.659402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-credential-socket\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.659742 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.659676 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-workload-socket\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.660042 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.659989 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-workload-certs\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.660042 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.660020 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-istio-data\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.660238 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.660045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-credential-socket\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.660238 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.660133 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26271269-74fa-4615-bb71-085a510fe223-istiod-ca-cert\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.661894 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.661874 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-istio-envoy\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.661986 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.661968 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26271269-74fa-4615-bb71-085a510fe223-istio-podinfo\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.670598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.670576 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26271269-74fa-4615-bb71-085a510fe223-istio-token\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.670969 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.670951 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krksw\" (UniqueName: \"kubernetes.io/projected/26271269-74fa-4615-bb71-085a510fe223-kube-api-access-krksw\") pod \"e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:11.853991 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:11.853909 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:12.139410 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:12.139382 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh"] Sep 08 10:19:12.140439 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:19:12.140417 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26271269_74fa_4615_bb71_085a510fe223.slice/crio-74dca75be1c53753533597c48e70f4a274ab8c29f47ee706593d35576eb03f17 WatchSource:0}: Error finding container 74dca75be1c53753533597c48e70f4a274ab8c29f47ee706593d35576eb03f17: Status 404 returned error can't find the container with id 74dca75be1c53753533597c48e70f4a274ab8c29f47ee706593d35576eb03f17 Sep 08 10:19:12.142488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:12.142451 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:19:12.142566 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:12.142521 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:19:12.142566 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:12.142550 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:19:12.467012 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:12.466979 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" event={"ID":"26271269-74fa-4615-bb71-085a510fe223","Type":"ContainerStarted","Data":"9bc2a5b6f0923d23161acd4dc7f085434fedaabbae50d0f609b41291fda55d84"} Sep 08 10:19:12.467198 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:12.467028 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" event={"ID":"26271269-74fa-4615-bb71-085a510fe223","Type":"ContainerStarted","Data":"74dca75be1c53753533597c48e70f4a274ab8c29f47ee706593d35576eb03f17"} Sep 08 10:19:12.491253 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:12.491195 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" podStartSLOduration=1.491175693 podStartE2EDuration="1.491175693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:19:12.488543429 +0000 UTC m=+1062.277595233" watchObservedRunningTime="2026-09-08 10:19:12.491175693 +0000 UTC m=+1062.280227494" Sep 08 10:19:12.854246 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:12.854148 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:13.728302 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:13.728266 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:13.826736 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:13.826705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:13.859436 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:13.859411 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:14.397014 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:14.396975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:14.477958 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:14.477926 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:14.478896 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:14.478879 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:14.490881 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:14.490811 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:15.600983 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.600948 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz"] Sep 08 10:19:15.601428 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.601389 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc97ae01-9034-497d-9b1b-e34c8997e485" containerName="istio-proxy" Sep 08 10:19:15.604545 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.604528 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:15.612219 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.612183 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w6wth\"" Sep 08 10:19:15.612369 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.612279 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-71a7a1-serving-cert\"" Sep 08 10:19:15.622508 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.622486 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz"] Sep 08 10:19:15.662927 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.662897 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:15.697874 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.697839 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:15.698070 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.697913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:15.698070 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.697955 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwbfv\" (UniqueName: \"kubernetes.io/projected/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-kube-api-access-cwbfv\") pod \"maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:15.698070 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.698034 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:15.783602 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.783571 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:15.798778 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.798748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:15.798974 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.798819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:15.798974 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.798863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:15.798974 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.798901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbfv\" (UniqueName: \"kubernetes.io/projected/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-kube-api-access-cwbfv\") pod \"maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:15.799783 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.799735 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:15.802252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.802224 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:15.802668 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.802645 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:15.808516 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.808493 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbfv\" (UniqueName: \"kubernetes.io/projected/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-kube-api-access-cwbfv\") pod \"maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:15.917537 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:15.917497 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:16.199284 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:16.199259 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz"] Sep 08 10:19:16.201931 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:19:16.201902 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6dc771_340e_4ee9_ad2f_e90c1aa12245.slice/crio-2508becd552d191539a2a64e73813394ab21f1b1f5887aa347da72fe33f7a09e WatchSource:0}: Error finding container 2508becd552d191539a2a64e73813394ab21f1b1f5887aa347da72fe33f7a09e: Status 404 returned error can't find the container with id 2508becd552d191539a2a64e73813394ab21f1b1f5887aa347da72fe33f7a09e Sep 08 10:19:16.203674 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:16.203655 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 10:19:16.491227 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:16.491136 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" event={"ID":"8c6dc771-340e-4ee9-ad2f-e90c1aa12245","Type":"ContainerStarted","Data":"2508becd552d191539a2a64e73813394ab21f1b1f5887aa347da72fe33f7a09e"} Sep 08 10:19:17.500560 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:17.500518 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" event={"ID":"8c6dc771-340e-4ee9-ad2f-e90c1aa12245","Type":"ContainerStarted","Data":"18f5ca181b0fdf9412427ee8dec9224e0087f2ba2ddc9450331f10dbef159b0b"} Sep 08 10:19:17.501055 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:17.500628 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:17.523106 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:17.523051 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" podStartSLOduration=2.160593527 podStartE2EDuration="2.523033167s" totalImagesPullingTime="362.43964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:19:17.520389157 +0000 UTC m=+1067.309440959" watchObservedRunningTime="2026-09-08 10:19:17.523033167 +0000 UTC m=+1067.312084970" Sep 08 10:19:17.878273 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:17.878187 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:17.972179 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:17.972150 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:18.390440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:18.390407 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:19.840394 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:19.840364 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:19.954509 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:19.954470 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:19:19.979951 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:19.979923 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:22.680048 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:22.680016 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:23.516643 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:23.516602 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:19:24.666331 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:24.666292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:24.765915 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:24.765878 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:25.166494 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:25.166461 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:25.620500 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:25.620413 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:25.973908 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:25.973879 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz"] Sep 08 10:19:25.977434 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:25.977417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:25.982178 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:25.982158 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-af58b2-openshift-default-dockercfg-n2xmz\"" Sep 08 10:19:25.999783 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:25.999751 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz"] Sep 08 10:19:26.012646 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.011306 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:26.101295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.101255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e3c867-ae25-4228-b754-3904bf18b88b-istio-podinfo\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.101565 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.101543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-workload-socket\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.101744 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.101722 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7z28\" (UniqueName: \"kubernetes.io/projected/43e3c867-ae25-4228-b754-3904bf18b88b-kube-api-access-d7z28\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.101922 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.101904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e3c867-ae25-4228-b754-3904bf18b88b-istiod-ca-cert\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.102055 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.102039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-workload-certs\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.102161 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.102145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-istio-data\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.102285 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.102269 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-credential-socket\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.102483 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.102464 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e3c867-ae25-4228-b754-3904bf18b88b-istio-token\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.102596 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.102580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-istio-envoy\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.203557 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.203516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-workload-socket\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.203836 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.203816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7z28\" (UniqueName: \"kubernetes.io/projected/43e3c867-ae25-4228-b754-3904bf18b88b-kube-api-access-d7z28\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.204002 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.203985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e3c867-ae25-4228-b754-3904bf18b88b-istiod-ca-cert\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.205542 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.205523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-workload-certs\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.205995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.205976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-istio-data\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.206403 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.206384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-credential-socket\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.206606 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.206589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e3c867-ae25-4228-b754-3904bf18b88b-istio-token\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.207250 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.207228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-istio-envoy\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.207754 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.207736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e3c867-ae25-4228-b754-3904bf18b88b-istio-podinfo\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.207925 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.206965 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-credential-socket\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.208036 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.204490 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-workload-socket\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.208131 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.205924 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-workload-certs\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.208223 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.205454 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e3c867-ae25-4228-b754-3904bf18b88b-istiod-ca-cert\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.208309 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.206333 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-istio-data\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.212067 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.212047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-istio-envoy\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.212692 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.212673 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e3c867-ae25-4228-b754-3904bf18b88b-istio-podinfo\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.216436 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.216414 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7z28\" (UniqueName: \"kubernetes.io/projected/43e3c867-ae25-4228-b754-3904bf18b88b-kube-api-access-d7z28\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.217763 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.217743 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e3c867-ae25-4228-b754-3904bf18b88b-istio-token\") pod \"e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.288367 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.288294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:26.320266 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.320238 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:26.586176 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.586152 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz"] Sep 08 10:19:26.587060 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:19:26.587035 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e3c867_ae25_4228_b754_3904bf18b88b.slice/crio-89b60f0e3c7f5be10dcedc20e569fdb4fcfbf71d05571f2f1996e963d2e7a097 WatchSource:0}: Error finding container 89b60f0e3c7f5be10dcedc20e569fdb4fcfbf71d05571f2f1996e963d2e7a097: Status 404 returned error can't find the container with id 89b60f0e3c7f5be10dcedc20e569fdb4fcfbf71d05571f2f1996e963d2e7a097 Sep 08 10:19:26.593825 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.593787 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:19:26.594055 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.594016 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:19:26.594236 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:26.594207 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:19:27.561107 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:27.561066 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" event={"ID":"43e3c867-ae25-4228-b754-3904bf18b88b","Type":"ContainerStarted","Data":"c1a9612a9e578021f7df393bcd1b4b5c56cea81afc4adfc34579b847e611d94a"} Sep 08 10:19:27.561506 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:27.561161 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" event={"ID":"43e3c867-ae25-4228-b754-3904bf18b88b","Type":"ContainerStarted","Data":"89b60f0e3c7f5be10dcedc20e569fdb4fcfbf71d05571f2f1996e963d2e7a097"} Sep 08 10:19:27.587556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:27.587500 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" podStartSLOduration=2.587484192 podStartE2EDuration="2.587484192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:19:27.585467228 +0000 UTC m=+1077.374519033" watchObservedRunningTime="2026-09-08 10:19:27.587484192 +0000 UTC m=+1077.376535996" Sep 08 10:19:28.289498 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:28.289458 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:28.295091 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:28.295063 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:28.567695 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:28.567587 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:28.568709 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:28.568691 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:29.576263 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:29.576227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:29.760074 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:29.760045 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:29.879800 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:29.879771 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": read tcp 10.134.0.2:51486->10.134.0.66:15021: read: connection reset by peer" start-of-body= Sep 08 10:19:29.879968 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:29.879876 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" podUID="26271269-74fa-4615-bb71-085a510fe223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": read tcp 10.134.0.2:51486->10.134.0.66:15021: read: connection reset by peer" Sep 08 10:19:30.581382 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:30.581351 2576 generic.go:356] "Generic (PLEG): container finished" podID="26271269-74fa-4615-bb71-085a510fe223" containerID="9bc2a5b6f0923d23161acd4dc7f085434fedaabbae50d0f609b41291fda55d84" exitCode=0 Sep 08 10:19:30.581796 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:30.581431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" event={"ID":"26271269-74fa-4615-bb71-085a510fe223","Type":"ContainerDied","Data":"9bc2a5b6f0923d23161acd4dc7f085434fedaabbae50d0f609b41291fda55d84"} Sep 08 10:19:30.581955 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:30.581936 2576 scope.go:117] "RemoveContainer" containerID="9bc2a5b6f0923d23161acd4dc7f085434fedaabbae50d0f609b41291fda55d84" Sep 08 10:19:30.582988 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:30.582954 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:19:30.583087 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:30.583025 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:19:30.583087 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:30.583060 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:19:30.901678 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:30.901643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:31.003893 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:31.003860 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:31.322648 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:31.322548 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:31.426947 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:31.426910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:31.510602 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:31.510569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:31.589720 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:31.589345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" event={"ID":"26271269-74fa-4615-bb71-085a510fe223","Type":"ContainerStarted","Data":"4f6de6638660118d155810c99ce06b6bd84141d71cc294b537d4bd5c4790f625"} Sep 08 10:19:31.854402 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:31.854313 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:31.854574 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:31.854522 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 08 10:19:31.854670 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:31.854584 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" podUID="26271269-74fa-4615-bb71-085a510fe223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 08 10:19:32.855259 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:32.855218 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 08 10:19:32.855742 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:32.855295 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" podUID="26271269-74fa-4615-bb71-085a510fe223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 08 10:19:33.854245 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:33.854212 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 08 10:19:33.854447 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:33.854272 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" podUID="26271269-74fa-4615-bb71-085a510fe223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 08 10:19:34.854810 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:34.854770 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 08 10:19:34.855219 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:34.854828 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" podUID="26271269-74fa-4615-bb71-085a510fe223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 08 10:19:35.854501 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:35.854468 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 08 10:19:35.854709 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:35.854537 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" podUID="26271269-74fa-4615-bb71-085a510fe223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 08 10:19:36.555975 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:36.555938 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:19:36.857918 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:36.857836 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:36.858341 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:36.858318 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:36.858919 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:36.858900 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:19:36.935069 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:36.935042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:39.396495 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:39.396452 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:39.765267 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:39.765190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:40.065412 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.065321 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j"] Sep 08 10:19:40.069829 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.069808 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.072344 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.072320 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d6e53c-openshift-default-dockercfg-j8fsn\"" Sep 08 10:19:40.082132 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.082111 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j"] Sep 08 10:19:40.110742 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.110714 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:40.156480 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.156441 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-workload-socket\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.156660 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.156491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdnfw\" (UniqueName: \"kubernetes.io/projected/871d1b48-9d6d-490e-aa12-14329ffef7a9-kube-api-access-fdnfw\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.156660 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.156560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-podinfo\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.156660 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.156602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871d1b48-9d6d-490e-aa12-14329ffef7a9-istiod-ca-cert\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.156660 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.156653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-workload-certs\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.156905 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.156692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-credential-socket\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.156905 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.156726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-data\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.156905 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.156824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-token\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.156905 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.156886 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-envoy\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.257643 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.257595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-workload-socket\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.257846 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.257673 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdnfw\" (UniqueName: \"kubernetes.io/projected/871d1b48-9d6d-490e-aa12-14329ffef7a9-kube-api-access-fdnfw\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.257846 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.257715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-podinfo\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.257846 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.257749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871d1b48-9d6d-490e-aa12-14329ffef7a9-istiod-ca-cert\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.257846 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.257774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-workload-certs\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.257846 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.257806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-credential-socket\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.258117 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.257853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-data\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.258117 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.257917 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-token\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.258117 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.257946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-envoy\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.258117 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.258006 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-workload-socket\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.258117 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.258086 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-workload-certs\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.258379 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.258242 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-credential-socket\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.258379 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.258331 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-data\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.258553 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.258531 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871d1b48-9d6d-490e-aa12-14329ffef7a9-istiod-ca-cert\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.260326 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.260298 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-envoy\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.260442 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.260426 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-podinfo\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.267364 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.267339 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-token\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.267544 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.267528 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdnfw\" (UniqueName: \"kubernetes.io/projected/871d1b48-9d6d-490e-aa12-14329ffef7a9-kube-api-access-fdnfw\") pod \"e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.381712 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.381609 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:40.725586 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.725554 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j"] Sep 08 10:19:40.726517 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:19:40.726492 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871d1b48_9d6d_490e_aa12_14329ffef7a9.slice/crio-8774dfa92947dd6202c365cc210b240810b4100eca3f9b66cc58e126a9c7a23c WatchSource:0}: Error finding container 8774dfa92947dd6202c365cc210b240810b4100eca3f9b66cc58e126a9c7a23c: Status 404 returned error can't find the container with id 8774dfa92947dd6202c365cc210b240810b4100eca3f9b66cc58e126a9c7a23c Sep 08 10:19:40.729340 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.729306 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:19:40.729453 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.729373 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:19:40.729453 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.729404 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:19:40.793637 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:40.793447 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:41.663239 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:41.663190 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" event={"ID":"871d1b48-9d6d-490e-aa12-14329ffef7a9","Type":"ContainerStarted","Data":"afe771bcd09a4750b054bbb8f306cd8678e1ab9d356b6542dc72253aab0e668d"} Sep 08 10:19:41.663423 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:41.663269 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" event={"ID":"871d1b48-9d6d-490e-aa12-14329ffef7a9","Type":"ContainerStarted","Data":"8774dfa92947dd6202c365cc210b240810b4100eca3f9b66cc58e126a9c7a23c"} Sep 08 10:19:41.686393 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:41.686349 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" podStartSLOduration=1.686335371 podStartE2EDuration="1.686335371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:19:41.686025985 +0000 UTC m=+1091.475077790" watchObservedRunningTime="2026-09-08 10:19:41.686335371 +0000 UTC m=+1091.475387175" Sep 08 10:19:42.382110 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:42.382071 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:42.387612 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:42.387586 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:42.390065 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:42.390040 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:42.675508 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:42.675461 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:42.682533 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:42.682508 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:19:43.682904 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:43.682876 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:43.804488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:43.804449 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:44.610788 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:44.610751 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 10:19:44.610980 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:44.610823 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" podUID="43e3c867-ae25-4228-b754-3904bf18b88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 10:19:44.852750 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:44.852713 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:47.365161 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:47.365123 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:47.510909 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:47.510871 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:47.709653 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:47.709605 2576 generic.go:356] "Generic (PLEG): container finished" podID="43e3c867-ae25-4228-b754-3904bf18b88b" containerID="c1a9612a9e578021f7df393bcd1b4b5c56cea81afc4adfc34579b847e611d94a" exitCode=0 Sep 08 10:19:47.709844 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:47.709677 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" event={"ID":"43e3c867-ae25-4228-b754-3904bf18b88b","Type":"ContainerDied","Data":"c1a9612a9e578021f7df393bcd1b4b5c56cea81afc4adfc34579b847e611d94a"} Sep 08 10:19:47.710086 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:47.710071 2576 scope.go:117] "RemoveContainer" containerID="c1a9612a9e578021f7df393bcd1b4b5c56cea81afc4adfc34579b847e611d94a" Sep 08 10:19:47.711213 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:47.711185 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:19:47.711321 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:47.711288 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:19:47.711383 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:47.711338 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:19:48.720821 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:48.720785 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" event={"ID":"43e3c867-ae25-4228-b754-3904bf18b88b","Type":"ContainerStarted","Data":"45a089731d0295032b19f898585280873b36d31f435f0e70ca14d8b1b5dfe97a"} Sep 08 10:19:49.289696 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:49.289172 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:49.289696 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:49.289326 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 08 10:19:49.289696 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:49.289371 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" podUID="43e3c867-ae25-4228-b754-3904bf18b88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 08 10:19:50.289677 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:50.289606 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 08 10:19:50.290240 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:50.290209 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" podUID="43e3c867-ae25-4228-b754-3904bf18b88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 08 10:19:51.289856 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:51.289820 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 08 10:19:51.290536 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:51.290502 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" podUID="43e3c867-ae25-4228-b754-3904bf18b88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 08 10:19:51.461171 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:51.461124 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:19:51.848220 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:51.848189 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:52.288833 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:52.288793 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 08 10:19:52.289153 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:52.289117 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" podUID="43e3c867-ae25-4228-b754-3904bf18b88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 08 10:19:53.182965 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:53.182923 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:53.289425 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:53.289386 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 08 10:19:53.289750 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:53.289717 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" podUID="43e3c867-ae25-4228-b754-3904bf18b88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 08 10:19:54.304249 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:54.304209 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:54.305475 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:54.305445 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:54.309015 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:54.308992 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:19:54.599489 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:54.599327 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:55.012050 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:55.012012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:55.340070 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:55.339994 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:58.860959 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:58.860931 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:59.035509 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:59.035472 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:19:59.113968 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:19:59.113877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:00.156955 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.156923 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb"] Sep 08 10:20:00.159952 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.159930 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:00.162264 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.162242 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-37ed05-serving-cert\"" Sep 08 10:20:00.170829 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.170800 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb"] Sep 08 10:20:00.267248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.267211 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7s2h\" (UniqueName: \"kubernetes.io/projected/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-kube-api-access-w7s2h\") pod \"maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:00.267418 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.267281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:00.267418 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.267344 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:00.267496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.267450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:00.349947 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.349904 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:00.368807 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.368778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:00.368959 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.368844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:00.369032 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.368963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7s2h\" (UniqueName: \"kubernetes.io/projected/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-kube-api-access-w7s2h\") pod \"maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:00.369032 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.369006 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:00.369603 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.369575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:00.371379 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.371356 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:00.371470 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.371403 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:00.378557 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.378534 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7s2h\" (UniqueName: \"kubernetes.io/projected/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-kube-api-access-w7s2h\") pod \"maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:00.472950 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.472848 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:00.475766 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.475737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:00.768127 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.768098 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb"] Sep 08 10:20:00.768741 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:20:00.768711 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2ecc57_e0a2_42c4_b1af_74a678d967c9.slice/crio-7afa1ff59ee15188ca40d11f800ffe60a052410e5f1ed5992e9954dc72d87930 WatchSource:0}: Error finding container 7afa1ff59ee15188ca40d11f800ffe60a052410e5f1ed5992e9954dc72d87930: Status 404 returned error can't find the container with id 7afa1ff59ee15188ca40d11f800ffe60a052410e5f1ed5992e9954dc72d87930 Sep 08 10:20:00.803880 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.803838 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" event={"ID":"8d2ecc57-e0a2-42c4-b1af-74a678d967c9","Type":"ContainerStarted","Data":"7afa1ff59ee15188ca40d11f800ffe60a052410e5f1ed5992e9954dc72d87930"} Sep 08 10:20:00.877730 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.877693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:00.981083 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:00.981049 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:01.812009 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:01.811966 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" event={"ID":"8d2ecc57-e0a2-42c4-b1af-74a678d967c9","Type":"ContainerStarted","Data":"8bea2c4404cd30916d2964b90e48d0d33c961974f9cec92571390d6dc26c5a65"} Sep 08 10:20:01.812399 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:01.812061 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:01.832628 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:01.832555 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" podStartSLOduration=1.26624728 podStartE2EDuration="1.832535578s" totalImagesPullingTime="566.288298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:20:01.828345394 +0000 UTC m=+1111.617397223" watchObservedRunningTime="2026-09-08 10:20:01.832535578 +0000 UTC m=+1111.621587396" Sep 08 10:20:02.820804 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:02.820775 2576 generic.go:356] "Generic (PLEG): container finished" podID="871d1b48-9d6d-490e-aa12-14329ffef7a9" containerID="afe771bcd09a4750b054bbb8f306cd8678e1ab9d356b6542dc72253aab0e668d" exitCode=0 Sep 08 10:20:02.821129 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:02.820842 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" event={"ID":"871d1b48-9d6d-490e-aa12-14329ffef7a9","Type":"ContainerDied","Data":"afe771bcd09a4750b054bbb8f306cd8678e1ab9d356b6542dc72253aab0e668d"} Sep 08 10:20:02.821382 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:02.821368 2576 scope.go:117] "RemoveContainer" containerID="afe771bcd09a4750b054bbb8f306cd8678e1ab9d356b6542dc72253aab0e668d" Sep 08 10:20:02.822341 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:02.822316 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:20:02.822440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:02.822380 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:20:02.822440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:02.822410 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:20:03.828510 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:03.828475 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" event={"ID":"871d1b48-9d6d-490e-aa12-14329ffef7a9","Type":"ContainerStarted","Data":"ad506f8e7a1c7dbd3742ad7410ecb43038a59e5cbd79f9a2496c48b5ab0999d3"} Sep 08 10:20:04.382463 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:04.382427 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:20:04.382725 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:04.382702 2576 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 08 10:20:04.382824 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:04.382762 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" podUID="871d1b48-9d6d-490e-aa12-14329ffef7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 08 10:20:05.382195 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:05.382158 2576 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 08 10:20:05.382588 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:05.382216 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" podUID="871d1b48-9d6d-490e-aa12-14329ffef7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 08 10:20:05.887187 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:05.887151 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:06.382234 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:06.382197 2576 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 08 10:20:06.382638 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:06.382272 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" podUID="871d1b48-9d6d-490e-aa12-14329ffef7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 08 10:20:07.382777 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:07.382736 2576 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 08 10:20:07.383252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:07.382799 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" podUID="871d1b48-9d6d-490e-aa12-14329ffef7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 08 10:20:07.826769 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:07.826731 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:20:08.382360 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:08.382325 2576 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 08 10:20:08.382538 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:08.382387 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" podUID="871d1b48-9d6d-490e-aa12-14329ffef7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 08 10:20:08.758860 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:08.758818 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:08.851068 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:08.851028 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:20:09.385969 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:09.385942 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:20:09.386371 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:09.386353 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:20:09.387004 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:09.386985 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:20:12.881195 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:12.881158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:20.056682 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:20.056649 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:20:21.424458 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:21.424418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:24.130875 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:24.130835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:24.636754 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:24.636721 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7887-140056932058720"] Sep 08 10:20:24.639902 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:24.639882 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932058720" Sep 08 10:20:24.642106 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:24.642087 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5nsxf\"" Sep 08 10:20:24.647065 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:24.647032 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7887-140056932058720"] Sep 08 10:20:24.714700 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:24.714664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lsk7\" (UniqueName: \"kubernetes.io/projected/960aae05-f7a5-4f95-9edf-345a4ffda998-kube-api-access-5lsk7\") pod \"test-curl-7887-140056932058720\" (UID: \"960aae05-f7a5-4f95-9edf-345a4ffda998\") " pod="opendatahub/test-curl-7887-140056932058720" Sep 08 10:20:24.815538 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:24.815503 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lsk7\" (UniqueName: \"kubernetes.io/projected/960aae05-f7a5-4f95-9edf-345a4ffda998-kube-api-access-5lsk7\") pod \"test-curl-7887-140056932058720\" (UID: \"960aae05-f7a5-4f95-9edf-345a4ffda998\") " pod="opendatahub/test-curl-7887-140056932058720" Sep 08 10:20:24.824709 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:24.824680 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lsk7\" (UniqueName: \"kubernetes.io/projected/960aae05-f7a5-4f95-9edf-345a4ffda998-kube-api-access-5lsk7\") pod \"test-curl-7887-140056932058720\" (UID: \"960aae05-f7a5-4f95-9edf-345a4ffda998\") " pod="opendatahub/test-curl-7887-140056932058720" Sep 08 10:20:24.950392 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:24.950354 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932058720" Sep 08 10:20:25.224252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:25.224225 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7887-140056932058720"] Sep 08 10:20:25.225814 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:20:25.225785 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod960aae05_f7a5_4f95_9edf_345a4ffda998.slice/crio-f306c085a0ef4ec89887ab6e313c74f9c6849469f0884a856e3e56ca1d4ddb8a WatchSource:0}: Error finding container f306c085a0ef4ec89887ab6e313c74f9c6849469f0884a856e3e56ca1d4ddb8a: Status 404 returned error can't find the container with id f306c085a0ef4ec89887ab6e313c74f9c6849469f0884a856e3e56ca1d4ddb8a Sep 08 10:20:25.961612 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:25.961580 2576 generic.go:356] "Generic (PLEG): container finished" podID="960aae05-f7a5-4f95-9edf-345a4ffda998" containerID="516cf5afa684fa045e09f65d3419ce5091181e8b2f6bb72ce3670eb1b8092a2c" exitCode=0 Sep 08 10:20:25.961811 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:25.961662 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7887-140056932058720" event={"ID":"960aae05-f7a5-4f95-9edf-345a4ffda998","Type":"ContainerDied","Data":"516cf5afa684fa045e09f65d3419ce5091181e8b2f6bb72ce3670eb1b8092a2c"} Sep 08 10:20:25.961811 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:25.961711 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7887-140056932058720" event={"ID":"960aae05-f7a5-4f95-9edf-345a4ffda998","Type":"ContainerStarted","Data":"f306c085a0ef4ec89887ab6e313c74f9c6849469f0884a856e3e56ca1d4ddb8a"} Sep 08 10:20:27.108946 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.108919 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932058720" Sep 08 10:20:27.131304 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.131282 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7887-140056932058720_960aae05-f7a5-4f95-9edf-345a4ffda998/test-curl-7887-140056932058720/0.log" Sep 08 10:20:27.139535 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.139510 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lsk7\" (UniqueName: \"kubernetes.io/projected/960aae05-f7a5-4f95-9edf-345a4ffda998-kube-api-access-5lsk7\") pod \"960aae05-f7a5-4f95-9edf-345a4ffda998\" (UID: \"960aae05-f7a5-4f95-9edf-345a4ffda998\") " Sep 08 10:20:27.141467 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.141441 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960aae05-f7a5-4f95-9edf-345a4ffda998-kube-api-access-5lsk7" (OuterVolumeSpecName: "kube-api-access-5lsk7") pod "960aae05-f7a5-4f95-9edf-345a4ffda998" (UID: "960aae05-f7a5-4f95-9edf-345a4ffda998"). InnerVolumeSpecName "kube-api-access-5lsk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:20:27.157413 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.157381 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7887-140056932058720"] Sep 08 10:20:27.159875 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.159855 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7887-140056932058720"] Sep 08 10:20:27.240469 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.240399 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lsk7\" (UniqueName: \"kubernetes.io/projected/960aae05-f7a5-4f95-9edf-345a4ffda998-kube-api-access-5lsk7\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:20:27.279027 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.279003 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7887-140056932058720"] Sep 08 10:20:27.279410 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.279399 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="960aae05-f7a5-4f95-9edf-345a4ffda998" containerName="test-curl-7887-140056932058720" Sep 08 10:20:27.279458 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.279412 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="960aae05-f7a5-4f95-9edf-345a4ffda998" containerName="test-curl-7887-140056932058720" Sep 08 10:20:27.279499 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.279490 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="960aae05-f7a5-4f95-9edf-345a4ffda998" containerName="test-curl-7887-140056932058720" Sep 08 10:20:27.281404 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.281389 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932058720" Sep 08 10:20:27.287198 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.287176 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7887-140056932058720"] Sep 08 10:20:27.341533 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.341510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8tgd\" (UniqueName: \"kubernetes.io/projected/1d7609d9-0048-48d1-93d4-82ffaf8013d9-kube-api-access-p8tgd\") pod \"test-curl-7887-140056932058720\" (UID: \"1d7609d9-0048-48d1-93d4-82ffaf8013d9\") " pod="opendatahub/test-curl-7887-140056932058720" Sep 08 10:20:27.442889 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.442853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8tgd\" (UniqueName: \"kubernetes.io/projected/1d7609d9-0048-48d1-93d4-82ffaf8013d9-kube-api-access-p8tgd\") pod \"test-curl-7887-140056932058720\" (UID: \"1d7609d9-0048-48d1-93d4-82ffaf8013d9\") " pod="opendatahub/test-curl-7887-140056932058720" Sep 08 10:20:27.451283 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.451260 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8tgd\" (UniqueName: \"kubernetes.io/projected/1d7609d9-0048-48d1-93d4-82ffaf8013d9-kube-api-access-p8tgd\") pod \"test-curl-7887-140056932058720\" (UID: \"1d7609d9-0048-48d1-93d4-82ffaf8013d9\") " pod="opendatahub/test-curl-7887-140056932058720" Sep 08 10:20:27.593044 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.592963 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932058720" Sep 08 10:20:27.862175 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.862148 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7887-140056932058720"] Sep 08 10:20:27.863173 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:20:27.863145 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d7609d9_0048_48d1_93d4_82ffaf8013d9.slice/crio-66a08294cd53f6057f3d31537a22080cdf1436d8bed30949c619ae9dcc5e0e8a WatchSource:0}: Error finding container 66a08294cd53f6057f3d31537a22080cdf1436d8bed30949c619ae9dcc5e0e8a: Status 404 returned error can't find the container with id 66a08294cd53f6057f3d31537a22080cdf1436d8bed30949c619ae9dcc5e0e8a Sep 08 10:20:27.974624 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.974598 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932058720" Sep 08 10:20:27.974941 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.974652 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f306c085a0ef4ec89887ab6e313c74f9c6849469f0884a856e3e56ca1d4ddb8a" Sep 08 10:20:27.975936 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.975913 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7887-140056932058720" event={"ID":"1d7609d9-0048-48d1-93d4-82ffaf8013d9","Type":"ContainerStarted","Data":"66a08294cd53f6057f3d31537a22080cdf1436d8bed30949c619ae9dcc5e0e8a"} Sep 08 10:20:27.991183 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:27.991154 2576 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7887-140056932058720" oldPodUID="960aae05-f7a5-4f95-9edf-345a4ffda998" podUID="1d7609d9-0048-48d1-93d4-82ffaf8013d9" Sep 08 10:20:28.678512 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:28.678480 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="960aae05-f7a5-4f95-9edf-345a4ffda998" path="/var/lib/kubelet/pods/960aae05-f7a5-4f95-9edf-345a4ffda998/volumes" Sep 08 10:20:28.983739 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:28.983702 2576 generic.go:356] "Generic (PLEG): container finished" podID="1d7609d9-0048-48d1-93d4-82ffaf8013d9" containerID="55158bece1fec21a6cb95d35ceaa25c82398c279592ec87db8b0625bcbd0be95" exitCode=0 Sep 08 10:20:28.983934 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:28.983802 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7887-140056932058720" event={"ID":"1d7609d9-0048-48d1-93d4-82ffaf8013d9","Type":"ContainerDied","Data":"55158bece1fec21a6cb95d35ceaa25c82398c279592ec87db8b0625bcbd0be95"} Sep 08 10:20:30.124075 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.124049 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932058720" Sep 08 10:20:30.144816 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.144794 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7887-140056932058720_1d7609d9-0048-48d1-93d4-82ffaf8013d9/test-curl-7887-140056932058720/0.log" Sep 08 10:20:30.167191 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.167152 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8tgd\" (UniqueName: \"kubernetes.io/projected/1d7609d9-0048-48d1-93d4-82ffaf8013d9-kube-api-access-p8tgd\") pod \"1d7609d9-0048-48d1-93d4-82ffaf8013d9\" (UID: \"1d7609d9-0048-48d1-93d4-82ffaf8013d9\") " Sep 08 10:20:30.167741 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.167716 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7887-140056932058720"] Sep 08 10:20:30.169551 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.169523 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7609d9-0048-48d1-93d4-82ffaf8013d9-kube-api-access-p8tgd" (OuterVolumeSpecName: "kube-api-access-p8tgd") pod "1d7609d9-0048-48d1-93d4-82ffaf8013d9" (UID: "1d7609d9-0048-48d1-93d4-82ffaf8013d9"). InnerVolumeSpecName "kube-api-access-p8tgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:20:30.171510 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.171489 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7887-140056932058720"] Sep 08 10:20:30.268731 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.268662 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8tgd\" (UniqueName: \"kubernetes.io/projected/1d7609d9-0048-48d1-93d4-82ffaf8013d9-kube-api-access-p8tgd\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:20:30.295964 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.295933 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7887-140056932059296"] Sep 08 10:20:30.296357 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.296344 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d7609d9-0048-48d1-93d4-82ffaf8013d9" containerName="test-curl-7887-140056932058720" Sep 08 10:20:30.296402 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.296359 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d7609d9-0048-48d1-93d4-82ffaf8013d9" containerName="test-curl-7887-140056932058720" Sep 08 10:20:30.296449 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.296433 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d7609d9-0048-48d1-93d4-82ffaf8013d9" containerName="test-curl-7887-140056932058720" Sep 08 10:20:30.298547 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.298530 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932059296" Sep 08 10:20:30.306080 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.306057 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7887-140056932059296"] Sep 08 10:20:30.369164 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.369129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfjmz\" (UniqueName: \"kubernetes.io/projected/9fed93fe-2349-4c9a-8ace-058f6e79848a-kube-api-access-gfjmz\") pod \"test-curl-7887-140056932059296\" (UID: \"9fed93fe-2349-4c9a-8ace-058f6e79848a\") " pod="opendatahub/test-curl-7887-140056932059296" Sep 08 10:20:30.469880 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.469841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfjmz\" (UniqueName: \"kubernetes.io/projected/9fed93fe-2349-4c9a-8ace-058f6e79848a-kube-api-access-gfjmz\") pod \"test-curl-7887-140056932059296\" (UID: \"9fed93fe-2349-4c9a-8ace-058f6e79848a\") " pod="opendatahub/test-curl-7887-140056932059296" Sep 08 10:20:30.479137 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.479107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfjmz\" (UniqueName: \"kubernetes.io/projected/9fed93fe-2349-4c9a-8ace-058f6e79848a-kube-api-access-gfjmz\") pod \"test-curl-7887-140056932059296\" (UID: \"9fed93fe-2349-4c9a-8ace-058f6e79848a\") " pod="opendatahub/test-curl-7887-140056932059296" Sep 08 10:20:30.610608 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.610531 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932059296" Sep 08 10:20:30.680910 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.680870 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d7609d9-0048-48d1-93d4-82ffaf8013d9" path="/var/lib/kubelet/pods/1d7609d9-0048-48d1-93d4-82ffaf8013d9/volumes" Sep 08 10:20:30.882966 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.882939 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7887-140056932059296"] Sep 08 10:20:30.884095 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:20:30.884071 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fed93fe_2349_4c9a_8ace_058f6e79848a.slice/crio-e77da0fdb5493afc8cae707cf310b0f27d7cf51f02a136ac5b30ee44302dec29 WatchSource:0}: Error finding container e77da0fdb5493afc8cae707cf310b0f27d7cf51f02a136ac5b30ee44302dec29: Status 404 returned error can't find the container with id e77da0fdb5493afc8cae707cf310b0f27d7cf51f02a136ac5b30ee44302dec29 Sep 08 10:20:30.997493 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.997463 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932058720" Sep 08 10:20:30.997713 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.997499 2576 scope.go:117] "RemoveContainer" containerID="55158bece1fec21a6cb95d35ceaa25c82398c279592ec87db8b0625bcbd0be95" Sep 08 10:20:30.999090 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:30.999050 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7887-140056932059296" event={"ID":"9fed93fe-2349-4c9a-8ace-058f6e79848a","Type":"ContainerStarted","Data":"e77da0fdb5493afc8cae707cf310b0f27d7cf51f02a136ac5b30ee44302dec29"} Sep 08 10:20:31.093048 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:31.093007 2576 scope.go:117] "RemoveContainer" containerID="f97888d5e844efff430d289e471d558f10b09de1eb50622946f84c4a30cc3c32" Sep 08 10:20:31.102679 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:31.102656 2576 scope.go:117] "RemoveContainer" containerID="6af6779fab9fd585be10c29b3d5bcf6515df8bd82b0fa9e0211b879b7786badb" Sep 08 10:20:32.007275 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:32.007241 2576 generic.go:356] "Generic (PLEG): container finished" podID="9fed93fe-2349-4c9a-8ace-058f6e79848a" containerID="1727c0af97e890f45cc36fc4624ca1f95942dd4736c5515898b97f3a84f8f4ab" exitCode=0 Sep 08 10:20:32.007662 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:32.007319 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7887-140056932059296" event={"ID":"9fed93fe-2349-4c9a-8ace-058f6e79848a","Type":"ContainerDied","Data":"1727c0af97e890f45cc36fc4624ca1f95942dd4736c5515898b97f3a84f8f4ab"} Sep 08 10:20:33.146432 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.146409 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932059296" Sep 08 10:20:33.169685 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.169662 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7887-140056932059296_9fed93fe-2349-4c9a-8ace-058f6e79848a/test-curl-7887-140056932059296/0.log" Sep 08 10:20:33.195296 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.195267 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfjmz\" (UniqueName: \"kubernetes.io/projected/9fed93fe-2349-4c9a-8ace-058f6e79848a-kube-api-access-gfjmz\") pod \"9fed93fe-2349-4c9a-8ace-058f6e79848a\" (UID: \"9fed93fe-2349-4c9a-8ace-058f6e79848a\") " Sep 08 10:20:33.195944 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.195919 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7887-140056932059296"] Sep 08 10:20:33.197562 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.197534 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fed93fe-2349-4c9a-8ace-058f6e79848a-kube-api-access-gfjmz" (OuterVolumeSpecName: "kube-api-access-gfjmz") pod "9fed93fe-2349-4c9a-8ace-058f6e79848a" (UID: "9fed93fe-2349-4c9a-8ace-058f6e79848a"). InnerVolumeSpecName "kube-api-access-gfjmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:20:33.200101 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.200084 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7887-140056932059296"] Sep 08 10:20:33.296787 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.296725 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfjmz\" (UniqueName: \"kubernetes.io/projected/9fed93fe-2349-4c9a-8ace-058f6e79848a-kube-api-access-gfjmz\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:20:33.327105 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.327078 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7887-140056932059296"] Sep 08 10:20:33.327498 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.327485 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fed93fe-2349-4c9a-8ace-058f6e79848a" containerName="test-curl-7887-140056932059296" Sep 08 10:20:33.327540 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.327499 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fed93fe-2349-4c9a-8ace-058f6e79848a" containerName="test-curl-7887-140056932059296" Sep 08 10:20:33.327583 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.327574 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fed93fe-2349-4c9a-8ace-058f6e79848a" containerName="test-curl-7887-140056932059296" Sep 08 10:20:33.330028 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.330013 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932059296" Sep 08 10:20:33.334985 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.334963 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7887-140056932059296"] Sep 08 10:20:33.397652 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.397611 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crp8r\" (UniqueName: \"kubernetes.io/projected/d6e4dfb6-2092-4034-ad07-337738751c87-kube-api-access-crp8r\") pod \"test-curl-7887-140056932059296\" (UID: \"d6e4dfb6-2092-4034-ad07-337738751c87\") " pod="opendatahub/test-curl-7887-140056932059296" Sep 08 10:20:33.498531 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.498505 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crp8r\" (UniqueName: \"kubernetes.io/projected/d6e4dfb6-2092-4034-ad07-337738751c87-kube-api-access-crp8r\") pod \"test-curl-7887-140056932059296\" (UID: \"d6e4dfb6-2092-4034-ad07-337738751c87\") " pod="opendatahub/test-curl-7887-140056932059296" Sep 08 10:20:33.512545 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.512515 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crp8r\" (UniqueName: \"kubernetes.io/projected/d6e4dfb6-2092-4034-ad07-337738751c87-kube-api-access-crp8r\") pod \"test-curl-7887-140056932059296\" (UID: \"d6e4dfb6-2092-4034-ad07-337738751c87\") " pod="opendatahub/test-curl-7887-140056932059296" Sep 08 10:20:33.640591 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.640565 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932059296" Sep 08 10:20:33.910633 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:33.910595 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7887-140056932059296"] Sep 08 10:20:33.911629 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:20:33.911593 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6e4dfb6_2092_4034_ad07_337738751c87.slice/crio-60f051d9a86d6e26efe8f414e67c025396c16ef63062235bba886aae40804f38 WatchSource:0}: Error finding container 60f051d9a86d6e26efe8f414e67c025396c16ef63062235bba886aae40804f38: Status 404 returned error can't find the container with id 60f051d9a86d6e26efe8f414e67c025396c16ef63062235bba886aae40804f38 Sep 08 10:20:34.021998 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:34.021964 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7887-140056932059296" event={"ID":"d6e4dfb6-2092-4034-ad07-337738751c87","Type":"ContainerStarted","Data":"60f051d9a86d6e26efe8f414e67c025396c16ef63062235bba886aae40804f38"} Sep 08 10:20:34.023255 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:34.023226 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932059296" Sep 08 10:20:34.023255 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:34.023245 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e77da0fdb5493afc8cae707cf310b0f27d7cf51f02a136ac5b30ee44302dec29" Sep 08 10:20:34.037040 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:34.037013 2576 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7887-140056932059296" oldPodUID="9fed93fe-2349-4c9a-8ace-058f6e79848a" podUID="d6e4dfb6-2092-4034-ad07-337738751c87" Sep 08 10:20:34.678539 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:34.678461 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fed93fe-2349-4c9a-8ace-058f6e79848a" path="/var/lib/kubelet/pods/9fed93fe-2349-4c9a-8ace-058f6e79848a/volumes" Sep 08 10:20:35.030390 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:35.030302 2576 generic.go:356] "Generic (PLEG): container finished" podID="d6e4dfb6-2092-4034-ad07-337738751c87" containerID="d41be29ff91c3db7097d17415fffb2ecec70f3757e7e3a4b952293a2e654c0fe" exitCode=0 Sep 08 10:20:35.030390 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:35.030348 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7887-140056932059296" event={"ID":"d6e4dfb6-2092-4034-ad07-337738751c87","Type":"ContainerDied","Data":"d41be29ff91c3db7097d17415fffb2ecec70f3757e7e3a4b952293a2e654c0fe"} Sep 08 10:20:36.170811 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:36.170790 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932059296" Sep 08 10:20:36.190372 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:36.190348 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7887-140056932059296_d6e4dfb6-2092-4034-ad07-337738751c87/test-curl-7887-140056932059296/0.log" Sep 08 10:20:36.218029 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:36.218005 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7887-140056932059296"] Sep 08 10:20:36.221745 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:36.221722 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7887-140056932059296"] Sep 08 10:20:36.226162 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:36.226143 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crp8r\" (UniqueName: \"kubernetes.io/projected/d6e4dfb6-2092-4034-ad07-337738751c87-kube-api-access-crp8r\") pod \"d6e4dfb6-2092-4034-ad07-337738751c87\" (UID: \"d6e4dfb6-2092-4034-ad07-337738751c87\") " Sep 08 10:20:36.228213 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:36.228191 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e4dfb6-2092-4034-ad07-337738751c87-kube-api-access-crp8r" (OuterVolumeSpecName: "kube-api-access-crp8r") pod "d6e4dfb6-2092-4034-ad07-337738751c87" (UID: "d6e4dfb6-2092-4034-ad07-337738751c87"). InnerVolumeSpecName "kube-api-access-crp8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:20:36.327052 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:36.326982 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crp8r\" (UniqueName: \"kubernetes.io/projected/d6e4dfb6-2092-4034-ad07-337738751c87-kube-api-access-crp8r\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:20:36.677928 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:36.677891 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6e4dfb6-2092-4034-ad07-337738751c87" path="/var/lib/kubelet/pods/d6e4dfb6-2092-4034-ad07-337738751c87/volumes" Sep 08 10:20:36.853814 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:36.853782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:37.044248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.044147 2576 scope.go:117] "RemoveContainer" containerID="d41be29ff91c3db7097d17415fffb2ecec70f3757e7e3a4b952293a2e654c0fe" Sep 08 10:20:37.044386 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.044169 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7887-140056932059296" Sep 08 10:20:37.148589 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.148560 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb"] Sep 08 10:20:37.149193 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.149139 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6e4dfb6-2092-4034-ad07-337738751c87" containerName="test-curl-7887-140056932059296" Sep 08 10:20:37.149193 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.149193 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6e4dfb6-2092-4034-ad07-337738751c87" containerName="test-curl-7887-140056932059296" Sep 08 10:20:37.149353 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.149333 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6e4dfb6-2092-4034-ad07-337738751c87" containerName="test-curl-7887-140056932059296" Sep 08 10:20:37.152343 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.152321 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.155018 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.154999 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8118c657-openshift-default-dockercfg-tvsrj\"" Sep 08 10:20:37.172007 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.171979 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb"] Sep 08 10:20:37.202690 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.202652 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:37.237430 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.237389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-token\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.237837 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.237778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-workload-socket\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.238071 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.238054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istiod-ca-cert\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.238262 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.238246 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-credential-socket\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.238457 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.238441 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-podinfo\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.238686 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.238669 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-envoy\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.238902 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.238884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzj7z\" (UniqueName: \"kubernetes.io/projected/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-kube-api-access-pzj7z\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.239098 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.239083 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-workload-certs\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.239273 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.239245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-data\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.334279 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.334189 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:37.340595 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.340571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istiod-ca-cert\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.340726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.340605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-credential-socket\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.340726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.340671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-podinfo\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.340726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.340693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-envoy\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.340874 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.340786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzj7z\" (UniqueName: \"kubernetes.io/projected/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-kube-api-access-pzj7z\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.340874 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.340852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-workload-certs\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.340986 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.340897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-data\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.340986 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.340931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-token\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.341083 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.340990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-workload-socket\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.341425 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.341244 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-credential-socket\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.341425 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.341298 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-workload-certs\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.341595 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.341464 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istiod-ca-cert\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.341595 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.341505 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-data\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.341595 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.341500 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-workload-socket\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.343222 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.343201 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-envoy\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.343331 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.343314 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-podinfo\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.349032 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.349013 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-token\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.349175 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.349158 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzj7z\" (UniqueName: \"kubernetes.io/projected/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-kube-api-access-pzj7z\") pod \"e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.463443 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.463412 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:37.764571 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.764544 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb"] Sep 08 10:20:37.764913 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:20:37.764893 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6f6be1_06d9_47e4_8c6f_9d4e13fa9cc0.slice/crio-5d41c255228facdf72befad3c81ee2121319b25828a78b222308d92a83b2e28c WatchSource:0}: Error finding container 5d41c255228facdf72befad3c81ee2121319b25828a78b222308d92a83b2e28c: Status 404 returned error can't find the container with id 5d41c255228facdf72befad3c81ee2121319b25828a78b222308d92a83b2e28c Sep 08 10:20:37.767587 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.767553 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:20:37.767695 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.767640 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:20:37.767695 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:37.767671 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:20:38.052645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:38.052539 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" event={"ID":"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0","Type":"ContainerStarted","Data":"826f8e4c739dbf61a3f7c7416d124c7e4b16aba7e276b9c91018ac2980c8e1e0"} Sep 08 10:20:38.052645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:38.052603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" event={"ID":"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0","Type":"ContainerStarted","Data":"5d41c255228facdf72befad3c81ee2121319b25828a78b222308d92a83b2e28c"} Sep 08 10:20:38.077158 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:38.077106 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" podStartSLOduration=1.077091582 podStartE2EDuration="1.077091582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:20:38.074090209 +0000 UTC m=+1147.863142049" watchObservedRunningTime="2026-09-08 10:20:38.077091582 +0000 UTC m=+1147.866143385" Sep 08 10:20:38.464039 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:38.464006 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:38.469005 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:38.468982 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:39.058396 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:39.058369 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:39.059217 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:39.059199 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:20:40.133319 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:40.133284 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:40.226791 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:40.226757 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:41.231060 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:41.230990 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:41.332269 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:41.332232 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:41.627930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:41.627847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:41.754949 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:41.754917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:41.894602 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:41.894545 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:20:48.959313 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:48.959278 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:20:55.910693 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:20:55.910548 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:21:00.202187 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:00.202156 2576 generic.go:356] "Generic (PLEG): container finished" podID="7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" containerID="826f8e4c739dbf61a3f7c7416d124c7e4b16aba7e276b9c91018ac2980c8e1e0" exitCode=0 Sep 08 10:21:00.202637 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:00.202237 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" event={"ID":"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0","Type":"ContainerDied","Data":"826f8e4c739dbf61a3f7c7416d124c7e4b16aba7e276b9c91018ac2980c8e1e0"} Sep 08 10:21:00.202761 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:00.202740 2576 scope.go:117] "RemoveContainer" containerID="826f8e4c739dbf61a3f7c7416d124c7e4b16aba7e276b9c91018ac2980c8e1e0" Sep 08 10:21:00.204089 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:00.204058 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:21:00.204188 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:00.204138 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:21:00.204251 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:00.204189 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:21:01.216887 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:01.216832 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" event={"ID":"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0","Type":"ContainerStarted","Data":"5cabb6a679c5ae44547572ae2c94fe99a853e74258fce3465f9d972b28ac2b12"} Sep 08 10:21:01.464289 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:01.464247 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:21:01.464523 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:01.464499 2576 patch_prober.go:28] interesting pod/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 08 10:21:01.464608 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:01.464560 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" podUID="7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 08 10:21:02.464263 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:02.464227 2576 patch_prober.go:28] interesting pod/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 08 10:21:02.464684 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:02.464284 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" podUID="7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 08 10:21:03.464166 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:03.464123 2576 patch_prober.go:28] interesting pod/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 08 10:21:03.464366 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:03.464193 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" podUID="7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 08 10:21:04.464801 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:04.464767 2576 patch_prober.go:28] interesting pod/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 08 10:21:04.465192 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:04.464822 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" podUID="7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 08 10:21:05.014388 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:05.014356 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:21:05.464774 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:05.464736 2576 patch_prober.go:28] interesting pod/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 08 10:21:05.464951 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:05.464796 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" podUID="7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 08 10:21:06.247653 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:06.247601 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:21:06.467938 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:06.467905 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:21:06.468325 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:06.468146 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:21:06.468843 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:06.468827 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:21:21.773101 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:21.773066 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:21:24.070820 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:24.070785 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:21:29.994921 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:29.994885 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:21:42.305589 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:42.305549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:21:46.157395 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:21:46.157358 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:22:00.293774 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:22:00.293742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:22:03.649490 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:22:03.649456 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:22:04.546443 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:22:04.546409 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:22:16.951216 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:22:16.951183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:22:17.052429 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:22:17.052395 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:22:19.539936 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:22:19.539902 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:22:24.624570 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:22:24.624446 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:22:25.744553 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:22:25.744520 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:22:43.053426 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:22:43.053381 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:22:58.851108 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:22:58.851078 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:23:01.720641 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:01.720580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:02.283101 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:02.283050 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:07.835390 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:07.835358 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:07.996169 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:07.996139 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:08.155853 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:08.155816 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb"] Sep 08 10:23:08.156119 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:08.156092 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" podUID="7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" containerName="istio-proxy" containerID="cri-o://5cabb6a679c5ae44547572ae2c94fe99a853e74258fce3465f9d972b28ac2b12" gracePeriod=30 Sep 08 10:23:08.179587 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:08.179552 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:09.718760 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:09.718726 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:09.818819 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:09.818746 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:09.918751 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:09.918715 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:10.633210 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:10.633179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:10.735275 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:10.735244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:10.836425 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:10.836394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:12.246097 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:12.246070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:13.407996 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.407972 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:23:13.502093 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502007 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzj7z\" (UniqueName: \"kubernetes.io/projected/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-kube-api-access-pzj7z\") pod \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " Sep 08 10:23:13.502093 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502045 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-data\") pod \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " Sep 08 10:23:13.502093 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502086 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-token\") pod \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " Sep 08 10:23:13.502367 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502133 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-envoy\") pod \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " Sep 08 10:23:13.502367 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502285 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-workload-certs\") pod \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " Sep 08 10:23:13.502367 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502346 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-credential-socket\") pod \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " Sep 08 10:23:13.502536 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502380 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" (UID: "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:23:13.502536 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502427 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-podinfo\") pod \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " Sep 08 10:23:13.502536 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502457 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istiod-ca-cert\") pod \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " Sep 08 10:23:13.502536 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502489 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-workload-socket\") pod \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\" (UID: \"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0\") " Sep 08 10:23:13.502536 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502520 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" (UID: "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:23:13.502814 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502596 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" (UID: "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:23:13.502814 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502795 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" (UID: "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:23:13.502814 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502801 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" (UID: "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:23:13.502974 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502946 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istiod-ca-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:23:13.502974 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502964 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-workload-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:23:13.503086 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502975 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-data\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:23:13.503086 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502984 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-workload-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:23:13.503086 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.502991 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-credential-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:23:13.504465 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.504435 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" (UID: "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:23:13.504578 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.504468 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" (UID: "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 10:23:13.504791 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.504772 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" (UID: "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:23:13.504791 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.504782 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-kube-api-access-pzj7z" (OuterVolumeSpecName: "kube-api-access-pzj7z") pod "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" (UID: "7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0"). InnerVolumeSpecName "kube-api-access-pzj7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:23:13.604143 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.604105 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-token\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:23:13.604143 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.604135 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-envoy\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:23:13.604143 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.604144 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-istio-podinfo\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:23:13.604143 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:13.604152 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzj7z\" (UniqueName: \"kubernetes.io/projected/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0-kube-api-access-pzj7z\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:23:14.027068 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:14.027032 2576 generic.go:356] "Generic (PLEG): container finished" podID="7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" containerID="5cabb6a679c5ae44547572ae2c94fe99a853e74258fce3465f9d972b28ac2b12" exitCode=0 Sep 08 10:23:14.027356 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:14.027078 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" Sep 08 10:23:14.027356 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:14.027123 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" event={"ID":"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0","Type":"ContainerDied","Data":"5cabb6a679c5ae44547572ae2c94fe99a853e74258fce3465f9d972b28ac2b12"} Sep 08 10:23:14.027356 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:14.027182 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb" event={"ID":"7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0","Type":"ContainerDied","Data":"5d41c255228facdf72befad3c81ee2121319b25828a78b222308d92a83b2e28c"} Sep 08 10:23:14.027356 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:14.027209 2576 scope.go:117] "RemoveContainer" containerID="5cabb6a679c5ae44547572ae2c94fe99a853e74258fce3465f9d972b28ac2b12" Sep 08 10:23:14.037032 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:14.037014 2576 scope.go:117] "RemoveContainer" containerID="826f8e4c739dbf61a3f7c7416d124c7e4b16aba7e276b9c91018ac2980c8e1e0" Sep 08 10:23:14.045945 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:14.045928 2576 scope.go:117] "RemoveContainer" containerID="5cabb6a679c5ae44547572ae2c94fe99a853e74258fce3465f9d972b28ac2b12" Sep 08 10:23:14.046235 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:23:14.046209 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cabb6a679c5ae44547572ae2c94fe99a853e74258fce3465f9d972b28ac2b12\": container with ID starting with 5cabb6a679c5ae44547572ae2c94fe99a853e74258fce3465f9d972b28ac2b12 not found: ID does not exist" containerID="5cabb6a679c5ae44547572ae2c94fe99a853e74258fce3465f9d972b28ac2b12" Sep 08 10:23:14.046333 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:14.046245 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cabb6a679c5ae44547572ae2c94fe99a853e74258fce3465f9d972b28ac2b12"} err="failed to get container status \"5cabb6a679c5ae44547572ae2c94fe99a853e74258fce3465f9d972b28ac2b12\": rpc error: code = NotFound desc = could not find container \"5cabb6a679c5ae44547572ae2c94fe99a853e74258fce3465f9d972b28ac2b12\": container with ID starting with 5cabb6a679c5ae44547572ae2c94fe99a853e74258fce3465f9d972b28ac2b12 not found: ID does not exist" Sep 08 10:23:14.046333 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:14.046265 2576 scope.go:117] "RemoveContainer" containerID="826f8e4c739dbf61a3f7c7416d124c7e4b16aba7e276b9c91018ac2980c8e1e0" Sep 08 10:23:14.046559 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:23:14.046531 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826f8e4c739dbf61a3f7c7416d124c7e4b16aba7e276b9c91018ac2980c8e1e0\": container with ID starting with 826f8e4c739dbf61a3f7c7416d124c7e4b16aba7e276b9c91018ac2980c8e1e0 not found: ID does not exist" containerID="826f8e4c739dbf61a3f7c7416d124c7e4b16aba7e276b9c91018ac2980c8e1e0" Sep 08 10:23:14.046936 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:14.046904 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826f8e4c739dbf61a3f7c7416d124c7e4b16aba7e276b9c91018ac2980c8e1e0"} err="failed to get container status \"826f8e4c739dbf61a3f7c7416d124c7e4b16aba7e276b9c91018ac2980c8e1e0\": rpc error: code = NotFound desc = could not find container \"826f8e4c739dbf61a3f7c7416d124c7e4b16aba7e276b9c91018ac2980c8e1e0\": container with ID starting with 826f8e4c739dbf61a3f7c7416d124c7e4b16aba7e276b9c91018ac2980c8e1e0 not found: ID does not exist" Sep 08 10:23:14.049682 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:14.049653 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb"] Sep 08 10:23:14.054281 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:14.054258 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8118c657-openshift-default-6467c8d684-t6bbb"] Sep 08 10:23:14.678458 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:14.678422 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" path="/var/lib/kubelet/pods/7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0/volumes" Sep 08 10:23:15.146002 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:15.145966 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:23:18.303719 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:18.303681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:18.405692 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:18.405657 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:19.080505 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:19.080476 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:24.907828 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:24.907790 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:25.026292 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:25.026259 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:25.118047 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:25.118015 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:25.886843 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:25.886808 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7890-dad3da"] Sep 08 10:23:25.887234 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:25.887221 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" containerName="istio-proxy" Sep 08 10:23:25.887285 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:25.887236 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" containerName="istio-proxy" Sep 08 10:23:25.887285 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:25.887244 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" containerName="istio-proxy" Sep 08 10:23:25.887285 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:25.887249 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" containerName="istio-proxy" Sep 08 10:23:25.887396 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:25.887311 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" containerName="istio-proxy" Sep 08 10:23:25.887396 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:25.887319 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e6f6be1-06d9-47e4-8c6f-9d4e13fa9cc0" containerName="istio-proxy" Sep 08 10:23:25.891947 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:25.891925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7890-dad3da" Sep 08 10:23:25.894592 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:25.894570 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5nsxf\"" Sep 08 10:23:25.898490 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:25.898470 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7890-dad3da"] Sep 08 10:23:25.924458 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:25.924432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6ks8\" (UniqueName: \"kubernetes.io/projected/71cda2b8-ad26-4dcb-b733-d130e6f09443-kube-api-access-s6ks8\") pod \"mt-curl-7890-dad3da\" (UID: \"71cda2b8-ad26-4dcb-b733-d130e6f09443\") " pod="opendatahub/mt-curl-7890-dad3da" Sep 08 10:23:26.025639 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:26.025588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6ks8\" (UniqueName: \"kubernetes.io/projected/71cda2b8-ad26-4dcb-b733-d130e6f09443-kube-api-access-s6ks8\") pod \"mt-curl-7890-dad3da\" (UID: \"71cda2b8-ad26-4dcb-b733-d130e6f09443\") " pod="opendatahub/mt-curl-7890-dad3da" Sep 08 10:23:26.044288 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:26.044257 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6ks8\" (UniqueName: \"kubernetes.io/projected/71cda2b8-ad26-4dcb-b733-d130e6f09443-kube-api-access-s6ks8\") pod \"mt-curl-7890-dad3da\" (UID: \"71cda2b8-ad26-4dcb-b733-d130e6f09443\") " pod="opendatahub/mt-curl-7890-dad3da" Sep 08 10:23:26.203707 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:26.203670 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7890-dad3da" Sep 08 10:23:26.475275 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:26.475252 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7890-dad3da"] Sep 08 10:23:26.477917 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:23:26.477885 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71cda2b8_ad26_4dcb_b733_d130e6f09443.slice/crio-f7c8352a3adcaa15d87262ca71e6962e8a9f0ce6448ebae7db771b64b3ef6ec5 WatchSource:0}: Error finding container f7c8352a3adcaa15d87262ca71e6962e8a9f0ce6448ebae7db771b64b3ef6ec5: Status 404 returned error can't find the container with id f7c8352a3adcaa15d87262ca71e6962e8a9f0ce6448ebae7db771b64b3ef6ec5 Sep 08 10:23:27.109714 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:27.109677 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7890-dad3da" event={"ID":"71cda2b8-ad26-4dcb-b733-d130e6f09443","Type":"ContainerStarted","Data":"f7c8352a3adcaa15d87262ca71e6962e8a9f0ce6448ebae7db771b64b3ef6ec5"} Sep 08 10:23:28.117053 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:28.117010 2576 generic.go:356] "Generic (PLEG): container finished" podID="71cda2b8-ad26-4dcb-b733-d130e6f09443" containerID="e976eb3f24810edfcbc93b7d2b525b4f86de0f7ece251806f78cd57312789f52" exitCode=0 Sep 08 10:23:28.117420 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:28.117099 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7890-dad3da" event={"ID":"71cda2b8-ad26-4dcb-b733-d130e6f09443","Type":"ContainerDied","Data":"e976eb3f24810edfcbc93b7d2b525b4f86de0f7ece251806f78cd57312789f52"} Sep 08 10:23:29.259912 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:29.259885 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7890-dad3da" Sep 08 10:23:29.280223 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:29.280196 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7890-dad3da_71cda2b8-ad26-4dcb-b733-d130e6f09443/mt-curl-7890-dad3da/0.log" Sep 08 10:23:29.304396 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:29.304366 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7890-dad3da"] Sep 08 10:23:29.308811 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:29.308787 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7890-dad3da"] Sep 08 10:23:29.356762 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:29.356735 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6ks8\" (UniqueName: \"kubernetes.io/projected/71cda2b8-ad26-4dcb-b733-d130e6f09443-kube-api-access-s6ks8\") pod \"71cda2b8-ad26-4dcb-b733-d130e6f09443\" (UID: \"71cda2b8-ad26-4dcb-b733-d130e6f09443\") " Sep 08 10:23:29.358794 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:29.358775 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cda2b8-ad26-4dcb-b733-d130e6f09443-kube-api-access-s6ks8" (OuterVolumeSpecName: "kube-api-access-s6ks8") pod "71cda2b8-ad26-4dcb-b733-d130e6f09443" (UID: "71cda2b8-ad26-4dcb-b733-d130e6f09443"). InnerVolumeSpecName "kube-api-access-s6ks8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:23:29.457599 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:29.457560 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6ks8\" (UniqueName: \"kubernetes.io/projected/71cda2b8-ad26-4dcb-b733-d130e6f09443-kube-api-access-s6ks8\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:23:30.131084 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:30.131052 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7890-dad3da" Sep 08 10:23:30.131292 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:30.131095 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7c8352a3adcaa15d87262ca71e6962e8a9f0ce6448ebae7db771b64b3ef6ec5" Sep 08 10:23:30.140452 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:23:30.140421 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7890-dad3da\" is forbidden: User \"system:node:ip-10-0-132-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-206.ec2.internal' and this object" podUID="71cda2b8-ad26-4dcb-b733-d130e6f09443" pod="opendatahub/mt-curl-7890-dad3da" Sep 08 10:23:30.146034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:30.146007 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:23:30.678944 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:23:30.678889 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7890-dad3da\" is forbidden: User \"system:node:ip-10-0-132-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-206.ec2.internal' and this object" podUID="71cda2b8-ad26-4dcb-b733-d130e6f09443" pod="opendatahub/mt-curl-7890-dad3da" Sep 08 10:23:30.681078 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:30.681047 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71cda2b8-ad26-4dcb-b733-d130e6f09443" path="/var/lib/kubelet/pods/71cda2b8-ad26-4dcb-b733-d130e6f09443/volumes" Sep 08 10:23:30.838181 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:30.838147 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:31.969760 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:31.969696 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:35.692593 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:35.692557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:35.962079 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:35.962002 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:23:43.262677 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:43.262638 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:23:59.322559 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:23:59.322482 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:02.667784 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:02.667752 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:02.768446 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:02.768407 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:03.062544 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.062465 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:03.381917 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.381833 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556"] Sep 08 10:24:03.382345 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.382330 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71cda2b8-ad26-4dcb-b733-d130e6f09443" containerName="mt-curl-7890-dad3da" Sep 08 10:24:03.382406 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.382347 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="71cda2b8-ad26-4dcb-b733-d130e6f09443" containerName="mt-curl-7890-dad3da" Sep 08 10:24:03.382453 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.382433 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="71cda2b8-ad26-4dcb-b733-d130e6f09443" containerName="mt-curl-7890-dad3da" Sep 08 10:24:03.385908 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.385887 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.388122 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.388103 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-22af43-openshift-default-dockercfg-z8qlb\"" Sep 08 10:24:03.398850 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.398830 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556"] Sep 08 10:24:03.414639 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.414589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:03.474050 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.474011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-credential-socket\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.474050 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.474051 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-workload-certs\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.474306 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.474102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istiod-ca-cert\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.474306 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.474122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-podinfo\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.474306 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.474198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-workload-socket\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.474306 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.474230 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpqb8\" (UniqueName: \"kubernetes.io/projected/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-kube-api-access-fpqb8\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.474306 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.474272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-envoy\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.474306 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.474296 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-data\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.474647 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.474319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-token\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.575207 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.575172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istiod-ca-cert\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.575207 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.575214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-podinfo\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.575440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.575250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-workload-socket\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.575440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.575273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpqb8\" (UniqueName: \"kubernetes.io/projected/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-kube-api-access-fpqb8\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.575440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.575328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-envoy\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.575440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.575359 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-data\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.575440 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.575382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-token\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.575735 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.575478 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-credential-socket\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.575735 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.575509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-workload-certs\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.575818 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.575768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-data\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.575870 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.575851 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-workload-certs\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.575925 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.575905 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-credential-socket\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.576023 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.576004 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-workload-socket\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.576074 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.576043 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istiod-ca-cert\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.577592 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.577563 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-envoy\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.577825 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.577807 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-podinfo\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.585334 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.585312 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-token\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.585683 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.585665 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpqb8\" (UniqueName: \"kubernetes.io/projected/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-kube-api-access-fpqb8\") pod \"e2e-api-a-22af43-openshift-default-7c4f46f4d-44556\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.633435 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.633367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:03.700505 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.700459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:03.980256 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.980226 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556"] Sep 08 10:24:03.981471 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:24:03.981444 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1f1345_2cdf_42b0_81ef_6a92a75a9c10.slice/crio-5f0907fd5f8bc6042fcf0b98273d78d37e443997a3d7e11778887aef55143c8b WatchSource:0}: Error finding container 5f0907fd5f8bc6042fcf0b98273d78d37e443997a3d7e11778887aef55143c8b: Status 404 returned error can't find the container with id 5f0907fd5f8bc6042fcf0b98273d78d37e443997a3d7e11778887aef55143c8b Sep 08 10:24:03.983710 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.983677 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:24:03.983779 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.983742 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:24:03.983779 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:03.983770 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:24:04.331942 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:04.331865 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" event={"ID":"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10","Type":"ContainerStarted","Data":"71837a327bbd7f1cbdbb7a0608558628e3730f21b49370e86db7f6ca2fb2c479"} Sep 08 10:24:04.331942 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:04.331914 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" event={"ID":"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10","Type":"ContainerStarted","Data":"5f0907fd5f8bc6042fcf0b98273d78d37e443997a3d7e11778887aef55143c8b"} Sep 08 10:24:04.353872 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:04.353824 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" podStartSLOduration=1.353808643 podStartE2EDuration="1.353808643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:24:04.350926841 +0000 UTC m=+1354.139978668" watchObservedRunningTime="2026-09-08 10:24:04.353808643 +0000 UTC m=+1354.142860446" Sep 08 10:24:04.700726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:04.700692 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:04.753030 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:04.753001 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:05.706309 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:05.706280 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:06.343444 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:06.343409 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:06.345064 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:06.345034 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:06.367939 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:06.367911 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:06.465735 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:06.465702 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:07.526735 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:07.526699 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:07.588721 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:07.588683 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:07.757799 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:07.757767 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:24:07.840954 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:07.840866 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:08.131924 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:08.131847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:08.236375 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:08.236345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:25.153815 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:25.153768 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:24:28.495920 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:28.495880 2576 generic.go:356] "Generic (PLEG): container finished" podID="bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" containerID="71837a327bbd7f1cbdbb7a0608558628e3730f21b49370e86db7f6ca2fb2c479" exitCode=0 Sep 08 10:24:28.496373 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:28.495960 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" event={"ID":"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10","Type":"ContainerDied","Data":"71837a327bbd7f1cbdbb7a0608558628e3730f21b49370e86db7f6ca2fb2c479"} Sep 08 10:24:28.496532 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:28.496512 2576 scope.go:117] "RemoveContainer" containerID="71837a327bbd7f1cbdbb7a0608558628e3730f21b49370e86db7f6ca2fb2c479" Sep 08 10:24:28.497810 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:28.497778 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:24:28.497918 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:28.497865 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:24:28.497984 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:28.497916 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:24:29.509231 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:29.509184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" event={"ID":"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10","Type":"ContainerStarted","Data":"dae7192bada8bd944f10263972f3e5567f3143979dafd3277aca3c78f5c1f191"} Sep 08 10:24:29.701608 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:29.701573 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:29.701928 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:29.701650 2576 patch_prober.go:28] interesting pod/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 08 10:24:29.702097 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:29.702074 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" podUID="bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 08 10:24:30.701830 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:30.701790 2576 patch_prober.go:28] interesting pod/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 08 10:24:30.702441 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:30.701861 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" podUID="bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 08 10:24:31.378402 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:31.378367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:31.701867 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:31.701827 2576 patch_prober.go:28] interesting pod/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 08 10:24:31.702464 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:31.702432 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" podUID="bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 08 10:24:31.858815 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:31.858790 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:32.150873 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.150834 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4"] Sep 08 10:24:32.153910 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.153885 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.158003 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.157961 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9e30f6-openshift-default-dockercfg-xcng8\"" Sep 08 10:24:32.177468 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.176716 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4"] Sep 08 10:24:32.224562 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.224528 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:32.276982 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.276941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-istio-data\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.277166 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.277031 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-credential-socket\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.277166 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.277078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99978507-8784-45cd-bd00-68f154580d0c-istio-podinfo\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.277166 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.277118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-workload-socket\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.277166 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.277149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99978507-8784-45cd-bd00-68f154580d0c-istio-token\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.277459 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.277186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-workload-certs\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.277459 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.277228 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6p79\" (UniqueName: \"kubernetes.io/projected/99978507-8784-45cd-bd00-68f154580d0c-kube-api-access-m6p79\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.277459 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.277261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-istio-envoy\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.277459 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.277366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99978507-8784-45cd-bd00-68f154580d0c-istiod-ca-cert\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.378589 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.378545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99978507-8784-45cd-bd00-68f154580d0c-istiod-ca-cert\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.378806 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.378607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-istio-data\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.378806 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.378662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-credential-socket\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.378806 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.378682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99978507-8784-45cd-bd00-68f154580d0c-istio-podinfo\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.378806 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.378706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-workload-socket\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.378806 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.378725 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99978507-8784-45cd-bd00-68f154580d0c-istio-token\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.378806 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.378749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-workload-certs\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.379151 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.379065 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-credential-socket\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.379151 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.379137 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-workload-certs\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.379245 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.379185 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-istio-data\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.379245 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.379194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6p79\" (UniqueName: \"kubernetes.io/projected/99978507-8784-45cd-bd00-68f154580d0c-kube-api-access-m6p79\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.379338 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.379247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-istio-envoy\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.379396 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.379331 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99978507-8784-45cd-bd00-68f154580d0c-istiod-ca-cert\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.379571 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.379544 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-workload-socket\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.381361 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.381342 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-istio-envoy\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.381464 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.381401 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99978507-8784-45cd-bd00-68f154580d0c-istio-podinfo\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.402123 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.402020 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6p79\" (UniqueName: \"kubernetes.io/projected/99978507-8784-45cd-bd00-68f154580d0c-kube-api-access-m6p79\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.404672 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.404635 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99978507-8784-45cd-bd00-68f154580d0c-istio-token\") pod \"e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.473241 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.473215 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:32.568224 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.568193 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:32.714698 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.701554 2576 patch_prober.go:28] interesting pod/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 08 10:24:32.714698 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.701642 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" podUID="bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 08 10:24:32.783381 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.783354 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4"] Sep 08 10:24:32.784571 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:24:32.784551 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99978507_8784_45cd_bd00_68f154580d0c.slice/crio-bcda4e7ad193a1f1ab3cae7d3929ac846c2a84a818b3c7019251823fc8e2bae8 WatchSource:0}: Error finding container bcda4e7ad193a1f1ab3cae7d3929ac846c2a84a818b3c7019251823fc8e2bae8: Status 404 returned error can't find the container with id bcda4e7ad193a1f1ab3cae7d3929ac846c2a84a818b3c7019251823fc8e2bae8 Sep 08 10:24:32.786865 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.786828 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:24:32.786970 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.786893 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:24:32.786970 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:32.786921 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:24:33.539698 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:33.539659 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" event={"ID":"99978507-8784-45cd-bd00-68f154580d0c","Type":"ContainerStarted","Data":"352ec1b36d9ebf21bd71ad6ca7915056cc6a63990a85465a13acda23d76ad8ae"} Sep 08 10:24:33.539904 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:33.539718 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" event={"ID":"99978507-8784-45cd-bd00-68f154580d0c","Type":"ContainerStarted","Data":"bcda4e7ad193a1f1ab3cae7d3929ac846c2a84a818b3c7019251823fc8e2bae8"} Sep 08 10:24:33.561376 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:33.561328 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" podStartSLOduration=1.561309042 podStartE2EDuration="1.561309042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:24:33.560168371 +0000 UTC m=+1383.349220177" watchObservedRunningTime="2026-09-08 10:24:33.561309042 +0000 UTC m=+1383.350360847" Sep 08 10:24:33.701585 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:33.701545 2576 patch_prober.go:28] interesting pod/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 08 10:24:33.701794 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:33.701607 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" podUID="bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 08 10:24:34.474007 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:34.473968 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:34.479224 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:34.479195 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:34.546244 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:34.546215 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:34.547424 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:34.547400 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:34.785562 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:34.785479 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:34.786157 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:34.786132 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:34.787542 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:34.787525 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:24:35.723173 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:35.723140 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:35.825876 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:35.825843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:36.654329 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.654295 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8"] Sep 08 10:24:36.671292 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.671255 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:36.676285 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.676261 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9e30f6-serving-cert\"" Sep 08 10:24:36.683277 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.683255 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8"] Sep 08 10:24:36.700653 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.700605 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:36.798931 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.798894 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:36.827152 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.827124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a4c803-536a-452c-b61a-4dd51d0b2831-metrics-certs\") pod \"maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:36.827284 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.827244 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a4c803-536a-452c-b61a-4dd51d0b2831-openshift-service-ca\") pod \"maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:36.827284 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.827278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q466n\" (UniqueName: \"kubernetes.io/projected/42a4c803-536a-452c-b61a-4dd51d0b2831-kube-api-access-q466n\") pod \"maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:36.827374 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.827307 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a4c803-536a-452c-b61a-4dd51d0b2831-maas-api-tls\") pod \"maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:36.927931 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.927858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q466n\" (UniqueName: \"kubernetes.io/projected/42a4c803-536a-452c-b61a-4dd51d0b2831-kube-api-access-q466n\") pod \"maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:36.927931 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.927903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a4c803-536a-452c-b61a-4dd51d0b2831-maas-api-tls\") pod \"maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:36.928130 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.928009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a4c803-536a-452c-b61a-4dd51d0b2831-metrics-certs\") pod \"maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:36.928130 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.928100 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a4c803-536a-452c-b61a-4dd51d0b2831-openshift-service-ca\") pod \"maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:36.928703 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.928680 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a4c803-536a-452c-b61a-4dd51d0b2831-openshift-service-ca\") pod \"maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:36.930330 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.930306 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a4c803-536a-452c-b61a-4dd51d0b2831-metrics-certs\") pod \"maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:36.930413 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.930356 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a4c803-536a-452c-b61a-4dd51d0b2831-maas-api-tls\") pod \"maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:36.935866 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.935847 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q466n\" (UniqueName: \"kubernetes.io/projected/42a4c803-536a-452c-b61a-4dd51d0b2831-kube-api-access-q466n\") pod \"maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:36.962139 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.962101 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:36.989803 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:36.989778 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:37.243888 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:37.243808 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:37.279675 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:37.279648 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8"] Sep 08 10:24:37.281354 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:24:37.281329 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a4c803_536a_452c_b61a_4dd51d0b2831.slice/crio-85ba870a6ae3a8c885694141d79af9a0df9056fb1c0420f922557576ec319cbf WatchSource:0}: Error finding container 85ba870a6ae3a8c885694141d79af9a0df9056fb1c0420f922557576ec319cbf: Status 404 returned error can't find the container with id 85ba870a6ae3a8c885694141d79af9a0df9056fb1c0420f922557576ec319cbf Sep 08 10:24:37.282876 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:37.282858 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 10:24:37.342538 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:37.342513 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:37.568836 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:37.568751 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" event={"ID":"42a4c803-536a-452c-b61a-4dd51d0b2831","Type":"ContainerStarted","Data":"85ba870a6ae3a8c885694141d79af9a0df9056fb1c0420f922557576ec319cbf"} Sep 08 10:24:38.257506 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:38.257474 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:38.576492 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:38.576398 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" event={"ID":"42a4c803-536a-452c-b61a-4dd51d0b2831","Type":"ContainerStarted","Data":"5c1056ff146de4dd77c2d7543afee053fd80012b2dc4ca80b51da726bf740f74"} Sep 08 10:24:38.576492 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:38.576463 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:38.596254 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:38.596203 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" podStartSLOduration=2.193823383 podStartE2EDuration="2.59618688s" totalImagesPullingTime="402.363497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:24:38.593254654 +0000 UTC m=+1388.382306472" watchObservedRunningTime="2026-09-08 10:24:38.59618688 +0000 UTC m=+1388.385238685" Sep 08 10:24:38.945304 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:38.945275 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:40.147864 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:40.147829 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:41.355458 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:41.355422 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:24:44.588502 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:44.588468 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:24:52.860171 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:52.860091 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:53.676435 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:53.676398 2576 generic.go:356] "Generic (PLEG): container finished" podID="99978507-8784-45cd-bd00-68f154580d0c" containerID="352ec1b36d9ebf21bd71ad6ca7915056cc6a63990a85465a13acda23d76ad8ae" exitCode=0 Sep 08 10:24:53.676720 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:53.676468 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" event={"ID":"99978507-8784-45cd-bd00-68f154580d0c","Type":"ContainerDied","Data":"352ec1b36d9ebf21bd71ad6ca7915056cc6a63990a85465a13acda23d76ad8ae"} Sep 08 10:24:53.676972 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:53.676955 2576 scope.go:117] "RemoveContainer" containerID="352ec1b36d9ebf21bd71ad6ca7915056cc6a63990a85465a13acda23d76ad8ae" Sep 08 10:24:53.678125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:53.678092 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:24:53.678212 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:53.678164 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:24:53.678272 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:53.678214 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:24:54.683842 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:54.683807 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" event={"ID":"99978507-8784-45cd-bd00-68f154580d0c","Type":"ContainerStarted","Data":"2671bae631cdb327294030d5eb63dd92fab0a446b91385b6b9f714d1c0c3d32b"} Sep 08 10:24:55.473792 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:55.473760 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:24:55.473957 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:55.473942 2576 patch_prober.go:28] interesting pod/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 08 10:24:55.474025 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:55.473987 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" podUID="99978507-8784-45cd-bd00-68f154580d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 08 10:24:56.474553 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:56.474512 2576 patch_prober.go:28] interesting pod/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 08 10:24:56.474983 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:56.474598 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" podUID="99978507-8784-45cd-bd00-68f154580d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 08 10:24:57.474732 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:57.474695 2576 patch_prober.go:28] interesting pod/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 08 10:24:57.475113 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:57.474747 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" podUID="99978507-8784-45cd-bd00-68f154580d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 08 10:24:57.682742 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:57.682708 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:24:58.474546 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:58.474516 2576 patch_prober.go:28] interesting pod/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 08 10:24:58.474817 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:58.474795 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" podUID="99978507-8784-45cd-bd00-68f154580d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 08 10:24:59.474037 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:59.473940 2576 patch_prober.go:28] interesting pod/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 08 10:24:59.474037 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:24:59.474029 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" podUID="99978507-8784-45cd-bd00-68f154580d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 08 10:25:00.478570 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:00.478539 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:25:00.479046 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:00.478975 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:25:00.479763 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:00.479736 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:25:00.521139 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:00.521106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:03.448355 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:03.448306 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:25:03.676407 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:03.676364 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:04.595036 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:04.595003 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:06.337373 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:06.337339 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:06.645547 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:06.645480 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:07.374427 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:07.374396 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:07.476782 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:07.476745 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:13.786831 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:13.786797 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:17.079716 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:17.079679 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:25:19.393807 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:19.393774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:19.494504 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:19.494473 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:19.594601 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:19.594557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:19.653519 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:19.653430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:19.961953 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:19.961871 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:22.339599 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:22.339565 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:25.827829 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:25.827748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:25.929869 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:25.929833 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:26.095237 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:26.095150 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556"] Sep 08 10:25:26.095402 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:26.095353 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" podUID="bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" containerName="istio-proxy" containerID="cri-o://dae7192bada8bd944f10263972f3e5567f3143979dafd3277aca3c78f5c1f191" gracePeriod=30 Sep 08 10:25:26.101246 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:26.101225 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:26.592437 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:26.592403 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:30.330938 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:30.330903 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7890-ab3399"] Sep 08 10:25:30.334242 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:30.334224 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7890-ab3399" Sep 08 10:25:30.336495 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:30.336471 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5nsxf\"" Sep 08 10:25:30.347977 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:30.347956 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7890-ab3399"] Sep 08 10:25:30.387512 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:30.387486 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvstn\" (UniqueName: \"kubernetes.io/projected/c4734a87-ec81-4955-8ab3-e571921aaa1f-kube-api-access-jvstn\") pod \"mt-curl-7890-ab3399\" (UID: \"c4734a87-ec81-4955-8ab3-e571921aaa1f\") " pod="opendatahub/mt-curl-7890-ab3399" Sep 08 10:25:30.488713 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:30.488681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvstn\" (UniqueName: \"kubernetes.io/projected/c4734a87-ec81-4955-8ab3-e571921aaa1f-kube-api-access-jvstn\") pod \"mt-curl-7890-ab3399\" (UID: \"c4734a87-ec81-4955-8ab3-e571921aaa1f\") " pod="opendatahub/mt-curl-7890-ab3399" Sep 08 10:25:30.497795 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:30.497774 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvstn\" (UniqueName: \"kubernetes.io/projected/c4734a87-ec81-4955-8ab3-e571921aaa1f-kube-api-access-jvstn\") pod \"mt-curl-7890-ab3399\" (UID: \"c4734a87-ec81-4955-8ab3-e571921aaa1f\") " pod="opendatahub/mt-curl-7890-ab3399" Sep 08 10:25:30.646028 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:30.646003 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5nsxf\"" Sep 08 10:25:30.653516 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:30.653499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7890-ab3399" Sep 08 10:25:30.929037 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:30.929010 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7890-ab3399"] Sep 08 10:25:30.931000 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:25:30.930976 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4734a87_ec81_4955_8ab3_e571921aaa1f.slice/crio-7ca6cdc3b6a8047a483839a6bc973a173df134dfd9ada944adb7a3b75616edbe WatchSource:0}: Error finding container 7ca6cdc3b6a8047a483839a6bc973a173df134dfd9ada944adb7a3b75616edbe: Status 404 returned error can't find the container with id 7ca6cdc3b6a8047a483839a6bc973a173df134dfd9ada944adb7a3b75616edbe Sep 08 10:25:31.371178 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.371155 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:25:31.397946 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.397916 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-data\") pod \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " Sep 08 10:25:31.398107 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.397975 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istiod-ca-cert\") pod \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " Sep 08 10:25:31.398107 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.398055 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpqb8\" (UniqueName: \"kubernetes.io/projected/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-kube-api-access-fpqb8\") pod \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " Sep 08 10:25:31.398238 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.398126 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-workload-certs\") pod \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " Sep 08 10:25:31.398238 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.398154 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-workload-socket\") pod \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " Sep 08 10:25:31.398238 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.398190 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-credential-socket\") pod \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " Sep 08 10:25:31.398238 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.398206 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-podinfo\") pod \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " Sep 08 10:25:31.398458 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.398249 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-token\") pod \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " Sep 08 10:25:31.398715 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.398683 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" (UID: "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:25:31.398835 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.398729 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" (UID: "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:25:31.398899 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.398856 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" (UID: "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:25:31.398899 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.398873 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-data" (OuterVolumeSpecName: "istio-data") pod "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" (UID: "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:25:31.399008 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.398926 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" (UID: "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:25:31.399113 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.399094 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-envoy\") pod \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\" (UID: \"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10\") " Sep 08 10:25:31.399520 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.399500 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-workload-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:31.399638 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.399524 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-workload-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:31.399638 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.399538 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-credential-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:31.399638 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.399552 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-data\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:31.399638 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.399566 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istiod-ca-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:31.401025 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.400993 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-kube-api-access-fpqb8" (OuterVolumeSpecName: "kube-api-access-fpqb8") pod "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" (UID: "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10"). InnerVolumeSpecName "kube-api-access-fpqb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:25:31.401129 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.401095 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" (UID: "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 10:25:31.401297 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.401269 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" (UID: "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:25:31.401553 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.401524 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-token" (OuterVolumeSpecName: "istio-token") pod "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" (UID: "bb1f1345-2cdf-42b0-81ef-6a92a75a9c10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:25:31.500975 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.500911 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-podinfo\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:31.500975 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.500936 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-token\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:31.500975 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.500945 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-istio-envoy\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:31.500975 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.500953 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpqb8\" (UniqueName: \"kubernetes.io/projected/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10-kube-api-access-fpqb8\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:31.931727 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.931688 2576 generic.go:356] "Generic (PLEG): container finished" podID="bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" containerID="dae7192bada8bd944f10263972f3e5567f3143979dafd3277aca3c78f5c1f191" exitCode=0 Sep 08 10:25:31.931919 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.931764 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" Sep 08 10:25:31.931919 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.931769 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" event={"ID":"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10","Type":"ContainerDied","Data":"dae7192bada8bd944f10263972f3e5567f3143979dafd3277aca3c78f5c1f191"} Sep 08 10:25:31.931919 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.931873 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556" event={"ID":"bb1f1345-2cdf-42b0-81ef-6a92a75a9c10","Type":"ContainerDied","Data":"5f0907fd5f8bc6042fcf0b98273d78d37e443997a3d7e11778887aef55143c8b"} Sep 08 10:25:31.931919 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.931908 2576 scope.go:117] "RemoveContainer" containerID="dae7192bada8bd944f10263972f3e5567f3143979dafd3277aca3c78f5c1f191" Sep 08 10:25:31.933453 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.933432 2576 generic.go:356] "Generic (PLEG): container finished" podID="c4734a87-ec81-4955-8ab3-e571921aaa1f" containerID="c5ef6d8c43c688f9d672364a835c575b97b5d0c62e7054f7368ba07425336112" exitCode=0 Sep 08 10:25:31.933537 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.933518 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7890-ab3399" event={"ID":"c4734a87-ec81-4955-8ab3-e571921aaa1f","Type":"ContainerDied","Data":"c5ef6d8c43c688f9d672364a835c575b97b5d0c62e7054f7368ba07425336112"} Sep 08 10:25:31.933597 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.933556 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7890-ab3399" event={"ID":"c4734a87-ec81-4955-8ab3-e571921aaa1f","Type":"ContainerStarted","Data":"7ca6cdc3b6a8047a483839a6bc973a173df134dfd9ada944adb7a3b75616edbe"} Sep 08 10:25:31.942746 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.942726 2576 scope.go:117] "RemoveContainer" containerID="71837a327bbd7f1cbdbb7a0608558628e3730f21b49370e86db7f6ca2fb2c479" Sep 08 10:25:31.951562 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.951546 2576 scope.go:117] "RemoveContainer" containerID="dae7192bada8bd944f10263972f3e5567f3143979dafd3277aca3c78f5c1f191" Sep 08 10:25:31.952028 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:25:31.952001 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae7192bada8bd944f10263972f3e5567f3143979dafd3277aca3c78f5c1f191\": container with ID starting with dae7192bada8bd944f10263972f3e5567f3143979dafd3277aca3c78f5c1f191 not found: ID does not exist" containerID="dae7192bada8bd944f10263972f3e5567f3143979dafd3277aca3c78f5c1f191" Sep 08 10:25:31.952184 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.952036 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae7192bada8bd944f10263972f3e5567f3143979dafd3277aca3c78f5c1f191"} err="failed to get container status \"dae7192bada8bd944f10263972f3e5567f3143979dafd3277aca3c78f5c1f191\": rpc error: code = NotFound desc = could not find container \"dae7192bada8bd944f10263972f3e5567f3143979dafd3277aca3c78f5c1f191\": container with ID starting with dae7192bada8bd944f10263972f3e5567f3143979dafd3277aca3c78f5c1f191 not found: ID does not exist" Sep 08 10:25:31.952184 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.952056 2576 scope.go:117] "RemoveContainer" containerID="71837a327bbd7f1cbdbb7a0608558628e3730f21b49370e86db7f6ca2fb2c479" Sep 08 10:25:31.952510 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:25:31.952491 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71837a327bbd7f1cbdbb7a0608558628e3730f21b49370e86db7f6ca2fb2c479\": container with ID starting with 71837a327bbd7f1cbdbb7a0608558628e3730f21b49370e86db7f6ca2fb2c479 not found: ID does not exist" containerID="71837a327bbd7f1cbdbb7a0608558628e3730f21b49370e86db7f6ca2fb2c479" Sep 08 10:25:31.952588 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.952518 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71837a327bbd7f1cbdbb7a0608558628e3730f21b49370e86db7f6ca2fb2c479"} err="failed to get container status \"71837a327bbd7f1cbdbb7a0608558628e3730f21b49370e86db7f6ca2fb2c479\": rpc error: code = NotFound desc = could not find container \"71837a327bbd7f1cbdbb7a0608558628e3730f21b49370e86db7f6ca2fb2c479\": container with ID starting with 71837a327bbd7f1cbdbb7a0608558628e3730f21b49370e86db7f6ca2fb2c479 not found: ID does not exist" Sep 08 10:25:31.966551 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.966531 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556"] Sep 08 10:25:31.970740 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:31.970719 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-22af43-openshift-default-7c4f46f4d-44556"] Sep 08 10:25:32.678757 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:32.678729 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" path="/var/lib/kubelet/pods/bb1f1345-2cdf-42b0-81ef-6a92a75a9c10/volumes" Sep 08 10:25:33.076538 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.076515 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7890-ab3399" Sep 08 10:25:33.098341 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.098315 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7890-ab3399_c4734a87-ec81-4955-8ab3-e571921aaa1f/mt-curl-7890-ab3399/0.log" Sep 08 10:25:33.118700 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.118676 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvstn\" (UniqueName: \"kubernetes.io/projected/c4734a87-ec81-4955-8ab3-e571921aaa1f-kube-api-access-jvstn\") pod \"c4734a87-ec81-4955-8ab3-e571921aaa1f\" (UID: \"c4734a87-ec81-4955-8ab3-e571921aaa1f\") " Sep 08 10:25:33.120825 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.120802 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4734a87-ec81-4955-8ab3-e571921aaa1f-kube-api-access-jvstn" (OuterVolumeSpecName: "kube-api-access-jvstn") pod "c4734a87-ec81-4955-8ab3-e571921aaa1f" (UID: "c4734a87-ec81-4955-8ab3-e571921aaa1f"). InnerVolumeSpecName "kube-api-access-jvstn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:25:33.124981 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.124959 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7890-ab3399"] Sep 08 10:25:33.130024 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.129997 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7890-ab3399"] Sep 08 10:25:33.220312 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.220241 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvstn\" (UniqueName: \"kubernetes.io/projected/c4734a87-ec81-4955-8ab3-e571921aaa1f-kube-api-access-jvstn\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:33.248362 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.248337 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7890-627e92"] Sep 08 10:25:33.248828 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.248814 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4734a87-ec81-4955-8ab3-e571921aaa1f" containerName="mt-curl-7890-ab3399" Sep 08 10:25:33.248875 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.248829 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4734a87-ec81-4955-8ab3-e571921aaa1f" containerName="mt-curl-7890-ab3399" Sep 08 10:25:33.248875 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.248841 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" containerName="istio-proxy" Sep 08 10:25:33.248875 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.248847 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" containerName="istio-proxy" Sep 08 10:25:33.248875 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.248862 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" containerName="istio-proxy" Sep 08 10:25:33.248875 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.248873 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" containerName="istio-proxy" Sep 08 10:25:33.249037 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.248942 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4734a87-ec81-4955-8ab3-e571921aaa1f" containerName="mt-curl-7890-ab3399" Sep 08 10:25:33.249037 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.248951 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" containerName="istio-proxy" Sep 08 10:25:33.252027 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.252012 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7890-627e92" Sep 08 10:25:33.259600 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.259573 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7890-627e92"] Sep 08 10:25:33.321821 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.321778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r8mk\" (UniqueName: \"kubernetes.io/projected/6e1487d6-5336-417f-abe2-02788e8c2253-kube-api-access-4r8mk\") pod \"mt-curl-7890-627e92\" (UID: \"6e1487d6-5336-417f-abe2-02788e8c2253\") " pod="opendatahub/mt-curl-7890-627e92" Sep 08 10:25:33.422977 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.422930 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8mk\" (UniqueName: \"kubernetes.io/projected/6e1487d6-5336-417f-abe2-02788e8c2253-kube-api-access-4r8mk\") pod \"mt-curl-7890-627e92\" (UID: \"6e1487d6-5336-417f-abe2-02788e8c2253\") " pod="opendatahub/mt-curl-7890-627e92" Sep 08 10:25:33.431209 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.431178 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r8mk\" (UniqueName: \"kubernetes.io/projected/6e1487d6-5336-417f-abe2-02788e8c2253-kube-api-access-4r8mk\") pod \"mt-curl-7890-627e92\" (UID: \"6e1487d6-5336-417f-abe2-02788e8c2253\") " pod="opendatahub/mt-curl-7890-627e92" Sep 08 10:25:33.445127 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.445100 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:25:33.562116 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.562030 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7890-627e92" Sep 08 10:25:33.835719 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.835686 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7890-627e92"] Sep 08 10:25:33.836381 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:25:33.836359 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e1487d6_5336_417f_abe2_02788e8c2253.slice/crio-6f9a5a27414bbab9663e4843cf9a4122896a8cdaa985bb59165264628684f50a WatchSource:0}: Error finding container 6f9a5a27414bbab9663e4843cf9a4122896a8cdaa985bb59165264628684f50a: Status 404 returned error can't find the container with id 6f9a5a27414bbab9663e4843cf9a4122896a8cdaa985bb59165264628684f50a Sep 08 10:25:33.948916 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.948876 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7890-627e92" event={"ID":"6e1487d6-5336-417f-abe2-02788e8c2253","Type":"ContainerStarted","Data":"6f9a5a27414bbab9663e4843cf9a4122896a8cdaa985bb59165264628684f50a"} Sep 08 10:25:33.950196 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.950176 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7890-ab3399" Sep 08 10:25:33.950309 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:33.950180 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ca6cdc3b6a8047a483839a6bc973a173df134dfd9ada944adb7a3b75616edbe" Sep 08 10:25:33.961039 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:25:33.961013 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7890-ab3399\" is forbidden: User \"system:node:ip-10-0-132-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-206.ec2.internal' and this object" podUID="c4734a87-ec81-4955-8ab3-e571921aaa1f" pod="opendatahub/mt-curl-7890-ab3399" Sep 08 10:25:34.678192 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:34.678160 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4734a87-ec81-4955-8ab3-e571921aaa1f" path="/var/lib/kubelet/pods/c4734a87-ec81-4955-8ab3-e571921aaa1f/volumes" Sep 08 10:25:34.957835 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:34.957744 2576 generic.go:356] "Generic (PLEG): container finished" podID="6e1487d6-5336-417f-abe2-02788e8c2253" containerID="08dd77138cb8a01b95404d574c8dd3cf4989887fe921dbc496722c5d81d623d7" exitCode=0 Sep 08 10:25:34.958180 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:34.957833 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7890-627e92" event={"ID":"6e1487d6-5336-417f-abe2-02788e8c2253","Type":"ContainerDied","Data":"08dd77138cb8a01b95404d574c8dd3cf4989887fe921dbc496722c5d81d623d7"} Sep 08 10:25:36.105111 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:36.105076 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7890-627e92" Sep 08 10:25:36.125082 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:36.125056 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7890-627e92_6e1487d6-5336-417f-abe2-02788e8c2253/mt-curl-7890-627e92/0.log" Sep 08 10:25:36.148366 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:36.148335 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7890-627e92"] Sep 08 10:25:36.151927 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:36.151903 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7890-627e92"] Sep 08 10:25:36.152195 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:36.152174 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r8mk\" (UniqueName: \"kubernetes.io/projected/6e1487d6-5336-417f-abe2-02788e8c2253-kube-api-access-4r8mk\") pod \"6e1487d6-5336-417f-abe2-02788e8c2253\" (UID: \"6e1487d6-5336-417f-abe2-02788e8c2253\") " Sep 08 10:25:36.154253 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:36.154230 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1487d6-5336-417f-abe2-02788e8c2253-kube-api-access-4r8mk" (OuterVolumeSpecName: "kube-api-access-4r8mk") pod "6e1487d6-5336-417f-abe2-02788e8c2253" (UID: "6e1487d6-5336-417f-abe2-02788e8c2253"). InnerVolumeSpecName "kube-api-access-4r8mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:25:36.253452 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:36.253372 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r8mk\" (UniqueName: \"kubernetes.io/projected/6e1487d6-5336-417f-abe2-02788e8c2253-kube-api-access-4r8mk\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:36.678055 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:36.678023 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e1487d6-5336-417f-abe2-02788e8c2253" path="/var/lib/kubelet/pods/6e1487d6-5336-417f-abe2-02788e8c2253/volumes" Sep 08 10:25:36.971050 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:36.970972 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7890-627e92" Sep 08 10:25:36.971050 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:36.971016 2576 scope.go:117] "RemoveContainer" containerID="08dd77138cb8a01b95404d574c8dd3cf4989887fe921dbc496722c5d81d623d7" Sep 08 10:25:38.219866 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:38.219834 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:39.195512 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:39.195477 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:41.895055 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:41.893912 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8"] Sep 08 10:25:41.895055 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:41.894180 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" podUID="42a4c803-536a-452c-b61a-4dd51d0b2831" containerName="maas-api" containerID="cri-o://5c1056ff146de4dd77c2d7543afee053fd80012b2dc4ca80b51da726bf740f74" gracePeriod=30 Sep 08 10:25:41.960254 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:41.960224 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:42.028688 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.028654 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:42.120249 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.120217 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:42.156063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.156004 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:25:42.191594 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.191555 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:42.213590 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.213548 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a4c803-536a-452c-b61a-4dd51d0b2831-maas-api-tls\") pod \"42a4c803-536a-452c-b61a-4dd51d0b2831\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " Sep 08 10:25:42.213801 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.213627 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a4c803-536a-452c-b61a-4dd51d0b2831-openshift-service-ca\") pod \"42a4c803-536a-452c-b61a-4dd51d0b2831\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " Sep 08 10:25:42.213801 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.213699 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a4c803-536a-452c-b61a-4dd51d0b2831-metrics-certs\") pod \"42a4c803-536a-452c-b61a-4dd51d0b2831\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " Sep 08 10:25:42.213801 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.213734 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q466n\" (UniqueName: \"kubernetes.io/projected/42a4c803-536a-452c-b61a-4dd51d0b2831-kube-api-access-q466n\") pod \"42a4c803-536a-452c-b61a-4dd51d0b2831\" (UID: \"42a4c803-536a-452c-b61a-4dd51d0b2831\") " Sep 08 10:25:42.214035 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.214009 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a4c803-536a-452c-b61a-4dd51d0b2831-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42a4c803-536a-452c-b61a-4dd51d0b2831" (UID: "42a4c803-536a-452c-b61a-4dd51d0b2831"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:25:42.215887 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.215855 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a4c803-536a-452c-b61a-4dd51d0b2831-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42a4c803-536a-452c-b61a-4dd51d0b2831" (UID: "42a4c803-536a-452c-b61a-4dd51d0b2831"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:25:42.216051 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.215902 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a4c803-536a-452c-b61a-4dd51d0b2831-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "42a4c803-536a-452c-b61a-4dd51d0b2831" (UID: "42a4c803-536a-452c-b61a-4dd51d0b2831"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:25:42.216051 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.215994 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a4c803-536a-452c-b61a-4dd51d0b2831-kube-api-access-q466n" (OuterVolumeSpecName: "kube-api-access-q466n") pod "42a4c803-536a-452c-b61a-4dd51d0b2831" (UID: "42a4c803-536a-452c-b61a-4dd51d0b2831"). InnerVolumeSpecName "kube-api-access-q466n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:25:42.315395 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.315352 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a4c803-536a-452c-b61a-4dd51d0b2831-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:42.315395 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.315392 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q466n\" (UniqueName: \"kubernetes.io/projected/42a4c803-536a-452c-b61a-4dd51d0b2831-kube-api-access-q466n\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:42.315729 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.315408 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a4c803-536a-452c-b61a-4dd51d0b2831-maas-api-tls\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:42.315729 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.315421 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a4c803-536a-452c-b61a-4dd51d0b2831-openshift-service-ca\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:42.488297 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:42.488216 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:43.011752 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:43.011720 2576 generic.go:356] "Generic (PLEG): container finished" podID="42a4c803-536a-452c-b61a-4dd51d0b2831" containerID="5c1056ff146de4dd77c2d7543afee053fd80012b2dc4ca80b51da726bf740f74" exitCode=0 Sep 08 10:25:43.012172 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:43.011767 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" Sep 08 10:25:43.012172 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:43.011810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" event={"ID":"42a4c803-536a-452c-b61a-4dd51d0b2831","Type":"ContainerDied","Data":"5c1056ff146de4dd77c2d7543afee053fd80012b2dc4ca80b51da726bf740f74"} Sep 08 10:25:43.012172 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:43.011861 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8" event={"ID":"42a4c803-536a-452c-b61a-4dd51d0b2831","Type":"ContainerDied","Data":"85ba870a6ae3a8c885694141d79af9a0df9056fb1c0420f922557576ec319cbf"} Sep 08 10:25:43.012172 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:43.011883 2576 scope.go:117] "RemoveContainer" containerID="5c1056ff146de4dd77c2d7543afee053fd80012b2dc4ca80b51da726bf740f74" Sep 08 10:25:43.021994 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:43.021975 2576 scope.go:117] "RemoveContainer" containerID="5c1056ff146de4dd77c2d7543afee053fd80012b2dc4ca80b51da726bf740f74" Sep 08 10:25:43.022304 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:25:43.022284 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c1056ff146de4dd77c2d7543afee053fd80012b2dc4ca80b51da726bf740f74\": container with ID starting with 5c1056ff146de4dd77c2d7543afee053fd80012b2dc4ca80b51da726bf740f74 not found: ID does not exist" containerID="5c1056ff146de4dd77c2d7543afee053fd80012b2dc4ca80b51da726bf740f74" Sep 08 10:25:43.022354 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:43.022315 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c1056ff146de4dd77c2d7543afee053fd80012b2dc4ca80b51da726bf740f74"} err="failed to get container status \"5c1056ff146de4dd77c2d7543afee053fd80012b2dc4ca80b51da726bf740f74\": rpc error: code = NotFound desc = could not find container \"5c1056ff146de4dd77c2d7543afee053fd80012b2dc4ca80b51da726bf740f74\": container with ID starting with 5c1056ff146de4dd77c2d7543afee053fd80012b2dc4ca80b51da726bf740f74 not found: ID does not exist" Sep 08 10:25:43.029863 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:43.029841 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8"] Sep 08 10:25:43.032328 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:43.032309 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9e30f6-8b76cb695-gpcq8"] Sep 08 10:25:44.678946 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:44.678915 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42a4c803-536a-452c-b61a-4dd51d0b2831" path="/var/lib/kubelet/pods/42a4c803-536a-452c-b61a-4dd51d0b2831/volumes" Sep 08 10:25:48.371316 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:48.371254 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:48.435744 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:48.435709 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:48.637073 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:48.636988 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4"] Sep 08 10:25:48.637319 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:48.637290 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" podUID="99978507-8784-45cd-bd00-68f154580d0c" containerName="istio-proxy" containerID="cri-o://2671bae631cdb327294030d5eb63dd92fab0a446b91385b6b9f714d1c0c3d32b" gracePeriod=30 Sep 08 10:25:48.654863 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:48.654831 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:49.284351 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:49.284318 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:52.268955 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:52.268912 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:25:53.898224 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:53.898196 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:25:54.035451 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.035351 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6p79\" (UniqueName: \"kubernetes.io/projected/99978507-8784-45cd-bd00-68f154580d0c-kube-api-access-m6p79\") pod \"99978507-8784-45cd-bd00-68f154580d0c\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " Sep 08 10:25:54.035451 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.035410 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-workload-certs\") pod \"99978507-8784-45cd-bd00-68f154580d0c\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " Sep 08 10:25:54.035451 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.035449 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99978507-8784-45cd-bd00-68f154580d0c-istiod-ca-cert\") pod \"99978507-8784-45cd-bd00-68f154580d0c\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " Sep 08 10:25:54.035806 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.035473 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-workload-socket\") pod \"99978507-8784-45cd-bd00-68f154580d0c\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " Sep 08 10:25:54.035806 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.035493 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-istio-data\") pod \"99978507-8784-45cd-bd00-68f154580d0c\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " Sep 08 10:25:54.035806 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.035517 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99978507-8784-45cd-bd00-68f154580d0c-istio-podinfo\") pod \"99978507-8784-45cd-bd00-68f154580d0c\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " Sep 08 10:25:54.035806 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.035564 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-credential-socket\") pod \"99978507-8784-45cd-bd00-68f154580d0c\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " Sep 08 10:25:54.035806 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.035593 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-istio-envoy\") pod \"99978507-8784-45cd-bd00-68f154580d0c\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " Sep 08 10:25:54.035806 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.035669 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99978507-8784-45cd-bd00-68f154580d0c-istio-token\") pod \"99978507-8784-45cd-bd00-68f154580d0c\" (UID: \"99978507-8784-45cd-bd00-68f154580d0c\") " Sep 08 10:25:54.036099 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.035850 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "99978507-8784-45cd-bd00-68f154580d0c" (UID: "99978507-8784-45cd-bd00-68f154580d0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:25:54.036099 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.035860 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "99978507-8784-45cd-bd00-68f154580d0c" (UID: "99978507-8784-45cd-bd00-68f154580d0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:25:54.036099 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.035875 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-istio-data" (OuterVolumeSpecName: "istio-data") pod "99978507-8784-45cd-bd00-68f154580d0c" (UID: "99978507-8784-45cd-bd00-68f154580d0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:25:54.036099 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.035922 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99978507-8784-45cd-bd00-68f154580d0c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "99978507-8784-45cd-bd00-68f154580d0c" (UID: "99978507-8784-45cd-bd00-68f154580d0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:25:54.036295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.036154 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "99978507-8784-45cd-bd00-68f154580d0c" (UID: "99978507-8784-45cd-bd00-68f154580d0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:25:54.036295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.036218 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-workload-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:54.036295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.036240 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99978507-8784-45cd-bd00-68f154580d0c-istiod-ca-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:54.036295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.036252 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-workload-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:54.036295 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.036265 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-istio-data\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:54.037887 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.037856 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99978507-8784-45cd-bd00-68f154580d0c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "99978507-8784-45cd-bd00-68f154580d0c" (UID: "99978507-8784-45cd-bd00-68f154580d0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 10:25:54.037887 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.037867 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99978507-8784-45cd-bd00-68f154580d0c-kube-api-access-m6p79" (OuterVolumeSpecName: "kube-api-access-m6p79") pod "99978507-8784-45cd-bd00-68f154580d0c" (UID: "99978507-8784-45cd-bd00-68f154580d0c"). InnerVolumeSpecName "kube-api-access-m6p79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:25:54.038273 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.038252 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "99978507-8784-45cd-bd00-68f154580d0c" (UID: "99978507-8784-45cd-bd00-68f154580d0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:25:54.038348 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.038284 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99978507-8784-45cd-bd00-68f154580d0c-istio-token" (OuterVolumeSpecName: "istio-token") pod "99978507-8784-45cd-bd00-68f154580d0c" (UID: "99978507-8784-45cd-bd00-68f154580d0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:25:54.084794 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.084762 2576 generic.go:356] "Generic (PLEG): container finished" podID="99978507-8784-45cd-bd00-68f154580d0c" containerID="2671bae631cdb327294030d5eb63dd92fab0a446b91385b6b9f714d1c0c3d32b" exitCode=0 Sep 08 10:25:54.084970 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.084825 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" Sep 08 10:25:54.084970 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.084855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" event={"ID":"99978507-8784-45cd-bd00-68f154580d0c","Type":"ContainerDied","Data":"2671bae631cdb327294030d5eb63dd92fab0a446b91385b6b9f714d1c0c3d32b"} Sep 08 10:25:54.084970 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.084939 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4" event={"ID":"99978507-8784-45cd-bd00-68f154580d0c","Type":"ContainerDied","Data":"bcda4e7ad193a1f1ab3cae7d3929ac846c2a84a818b3c7019251823fc8e2bae8"} Sep 08 10:25:54.084970 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.084963 2576 scope.go:117] "RemoveContainer" containerID="2671bae631cdb327294030d5eb63dd92fab0a446b91385b6b9f714d1c0c3d32b" Sep 08 10:25:54.096341 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.096319 2576 scope.go:117] "RemoveContainer" containerID="352ec1b36d9ebf21bd71ad6ca7915056cc6a63990a85465a13acda23d76ad8ae" Sep 08 10:25:54.105684 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.105662 2576 scope.go:117] "RemoveContainer" containerID="2671bae631cdb327294030d5eb63dd92fab0a446b91385b6b9f714d1c0c3d32b" Sep 08 10:25:54.105991 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:25:54.105970 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2671bae631cdb327294030d5eb63dd92fab0a446b91385b6b9f714d1c0c3d32b\": container with ID starting with 2671bae631cdb327294030d5eb63dd92fab0a446b91385b6b9f714d1c0c3d32b not found: ID does not exist" containerID="2671bae631cdb327294030d5eb63dd92fab0a446b91385b6b9f714d1c0c3d32b" Sep 08 10:25:54.106069 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.106006 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2671bae631cdb327294030d5eb63dd92fab0a446b91385b6b9f714d1c0c3d32b"} err="failed to get container status \"2671bae631cdb327294030d5eb63dd92fab0a446b91385b6b9f714d1c0c3d32b\": rpc error: code = NotFound desc = could not find container \"2671bae631cdb327294030d5eb63dd92fab0a446b91385b6b9f714d1c0c3d32b\": container with ID starting with 2671bae631cdb327294030d5eb63dd92fab0a446b91385b6b9f714d1c0c3d32b not found: ID does not exist" Sep 08 10:25:54.106069 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.106032 2576 scope.go:117] "RemoveContainer" containerID="352ec1b36d9ebf21bd71ad6ca7915056cc6a63990a85465a13acda23d76ad8ae" Sep 08 10:25:54.106305 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:25:54.106280 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352ec1b36d9ebf21bd71ad6ca7915056cc6a63990a85465a13acda23d76ad8ae\": container with ID starting with 352ec1b36d9ebf21bd71ad6ca7915056cc6a63990a85465a13acda23d76ad8ae not found: ID does not exist" containerID="352ec1b36d9ebf21bd71ad6ca7915056cc6a63990a85465a13acda23d76ad8ae" Sep 08 10:25:54.106406 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.106309 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352ec1b36d9ebf21bd71ad6ca7915056cc6a63990a85465a13acda23d76ad8ae"} err="failed to get container status \"352ec1b36d9ebf21bd71ad6ca7915056cc6a63990a85465a13acda23d76ad8ae\": rpc error: code = NotFound desc = could not find container \"352ec1b36d9ebf21bd71ad6ca7915056cc6a63990a85465a13acda23d76ad8ae\": container with ID starting with 352ec1b36d9ebf21bd71ad6ca7915056cc6a63990a85465a13acda23d76ad8ae not found: ID does not exist" Sep 08 10:25:54.111094 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.111068 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4"] Sep 08 10:25:54.113761 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.113737 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9e30f6-openshift-default-dffdf854d-4cll4"] Sep 08 10:25:54.137280 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.137247 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99978507-8784-45cd-bd00-68f154580d0c-istio-token\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:54.137280 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.137273 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6p79\" (UniqueName: \"kubernetes.io/projected/99978507-8784-45cd-bd00-68f154580d0c-kube-api-access-m6p79\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:54.137280 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.137285 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99978507-8784-45cd-bd00-68f154580d0c-istio-podinfo\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:54.137515 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.137295 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-credential-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:54.137515 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.137304 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99978507-8784-45cd-bd00-68f154580d0c-istio-envoy\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:25:54.678463 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.678427 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99978507-8784-45cd-bd00-68f154580d0c" path="/var/lib/kubelet/pods/99978507-8784-45cd-bd00-68f154580d0c/volumes" Sep 08 10:25:54.934089 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:54.934001 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:59.542870 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:59.542834 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:59.642635 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:59.642599 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:59.725181 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:59.725147 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:25:59.786978 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:25:59.786944 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:00.135951 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:00.135919 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:05.937209 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:05.937175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:06.007074 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:06.007040 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:06.268012 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:06.267920 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz"] Sep 08 10:26:06.268167 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:06.268137 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" podUID="43e3c867-ae25-4228-b754-3904bf18b88b" containerName="istio-proxy" containerID="cri-o://45a089731d0295032b19f898585280873b36d31f435f0e70ca14d8b1b5dfe97a" gracePeriod=30 Sep 08 10:26:06.271218 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:06.271194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:06.690416 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:06.690384 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:06.749204 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:06.749166 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:26:09.308032 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:09.307991 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 10:26:09.308519 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:09.308067 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" podUID="43e3c867-ae25-4228-b754-3904bf18b88b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 10:26:11.530059 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.529989 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:26:11.614333 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.614290 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-istio-envoy\") pod \"43e3c867-ae25-4228-b754-3904bf18b88b\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " Sep 08 10:26:11.614333 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.614343 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-workload-certs\") pod \"43e3c867-ae25-4228-b754-3904bf18b88b\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " Sep 08 10:26:11.614594 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.614365 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-workload-socket\") pod \"43e3c867-ae25-4228-b754-3904bf18b88b\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " Sep 08 10:26:11.614594 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.614427 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e3c867-ae25-4228-b754-3904bf18b88b-istio-podinfo\") pod \"43e3c867-ae25-4228-b754-3904bf18b88b\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " Sep 08 10:26:11.614594 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.614448 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-credential-socket\") pod \"43e3c867-ae25-4228-b754-3904bf18b88b\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " Sep 08 10:26:11.614594 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.614491 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7z28\" (UniqueName: \"kubernetes.io/projected/43e3c867-ae25-4228-b754-3904bf18b88b-kube-api-access-d7z28\") pod \"43e3c867-ae25-4228-b754-3904bf18b88b\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " Sep 08 10:26:11.614594 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.614578 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e3c867-ae25-4228-b754-3904bf18b88b-istiod-ca-cert\") pod \"43e3c867-ae25-4228-b754-3904bf18b88b\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " Sep 08 10:26:11.614910 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.614602 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e3c867-ae25-4228-b754-3904bf18b88b-istio-token\") pod \"43e3c867-ae25-4228-b754-3904bf18b88b\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " Sep 08 10:26:11.614910 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.614677 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "43e3c867-ae25-4228-b754-3904bf18b88b" (UID: "43e3c867-ae25-4228-b754-3904bf18b88b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:26:11.614910 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.614745 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "43e3c867-ae25-4228-b754-3904bf18b88b" (UID: "43e3c867-ae25-4228-b754-3904bf18b88b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:26:11.614910 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.614873 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-istio-data\") pod \"43e3c867-ae25-4228-b754-3904bf18b88b\" (UID: \"43e3c867-ae25-4228-b754-3904bf18b88b\") " Sep 08 10:26:11.614910 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.614869 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "43e3c867-ae25-4228-b754-3904bf18b88b" (UID: "43e3c867-ae25-4228-b754-3904bf18b88b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:26:11.615218 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.615042 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e3c867-ae25-4228-b754-3904bf18b88b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "43e3c867-ae25-4228-b754-3904bf18b88b" (UID: "43e3c867-ae25-4228-b754-3904bf18b88b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:26:11.615276 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.615235 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-credential-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:26:11.615368 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.615344 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-istio-data" (OuterVolumeSpecName: "istio-data") pod "43e3c867-ae25-4228-b754-3904bf18b88b" (UID: "43e3c867-ae25-4228-b754-3904bf18b88b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:26:11.615420 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.615357 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e3c867-ae25-4228-b754-3904bf18b88b-istiod-ca-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:26:11.615420 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.615399 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-workload-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:26:11.615490 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.615416 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-workload-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:26:11.616655 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.616596 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "43e3c867-ae25-4228-b754-3904bf18b88b" (UID: "43e3c867-ae25-4228-b754-3904bf18b88b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:26:11.616767 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.616718 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e3c867-ae25-4228-b754-3904bf18b88b-istio-token" (OuterVolumeSpecName: "istio-token") pod "43e3c867-ae25-4228-b754-3904bf18b88b" (UID: "43e3c867-ae25-4228-b754-3904bf18b88b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:26:11.617124 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.617102 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e3c867-ae25-4228-b754-3904bf18b88b-kube-api-access-d7z28" (OuterVolumeSpecName: "kube-api-access-d7z28") pod "43e3c867-ae25-4228-b754-3904bf18b88b" (UID: "43e3c867-ae25-4228-b754-3904bf18b88b"). InnerVolumeSpecName "kube-api-access-d7z28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:26:11.617124 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.617110 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43e3c867-ae25-4228-b754-3904bf18b88b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "43e3c867-ae25-4228-b754-3904bf18b88b" (UID: "43e3c867-ae25-4228-b754-3904bf18b88b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 10:26:11.716504 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.716466 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e3c867-ae25-4228-b754-3904bf18b88b-istio-podinfo\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:26:11.716504 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.716498 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7z28\" (UniqueName: \"kubernetes.io/projected/43e3c867-ae25-4228-b754-3904bf18b88b-kube-api-access-d7z28\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:26:11.716504 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.716509 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e3c867-ae25-4228-b754-3904bf18b88b-istio-token\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:26:11.716504 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.716517 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-istio-data\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:26:11.716818 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:11.716525 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e3c867-ae25-4228-b754-3904bf18b88b-istio-envoy\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:26:12.201717 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:12.201678 2576 generic.go:356] "Generic (PLEG): container finished" podID="43e3c867-ae25-4228-b754-3904bf18b88b" containerID="45a089731d0295032b19f898585280873b36d31f435f0e70ca14d8b1b5dfe97a" exitCode=0 Sep 08 10:26:12.201717 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:12.201730 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" Sep 08 10:26:12.201988 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:12.201764 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" event={"ID":"43e3c867-ae25-4228-b754-3904bf18b88b","Type":"ContainerDied","Data":"45a089731d0295032b19f898585280873b36d31f435f0e70ca14d8b1b5dfe97a"} Sep 08 10:26:12.201988 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:12.201833 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz" event={"ID":"43e3c867-ae25-4228-b754-3904bf18b88b","Type":"ContainerDied","Data":"89b60f0e3c7f5be10dcedc20e569fdb4fcfbf71d05571f2f1996e963d2e7a097"} Sep 08 10:26:12.201988 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:12.201865 2576 scope.go:117] "RemoveContainer" containerID="45a089731d0295032b19f898585280873b36d31f435f0e70ca14d8b1b5dfe97a" Sep 08 10:26:12.212252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:12.212235 2576 scope.go:117] "RemoveContainer" containerID="c1a9612a9e578021f7df393bcd1b4b5c56cea81afc4adfc34579b847e611d94a" Sep 08 10:26:12.221366 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:12.221343 2576 scope.go:117] "RemoveContainer" containerID="45a089731d0295032b19f898585280873b36d31f435f0e70ca14d8b1b5dfe97a" Sep 08 10:26:12.221719 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:26:12.221699 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a089731d0295032b19f898585280873b36d31f435f0e70ca14d8b1b5dfe97a\": container with ID starting with 45a089731d0295032b19f898585280873b36d31f435f0e70ca14d8b1b5dfe97a not found: ID does not exist" containerID="45a089731d0295032b19f898585280873b36d31f435f0e70ca14d8b1b5dfe97a" Sep 08 10:26:12.221771 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:12.221731 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a089731d0295032b19f898585280873b36d31f435f0e70ca14d8b1b5dfe97a"} err="failed to get container status \"45a089731d0295032b19f898585280873b36d31f435f0e70ca14d8b1b5dfe97a\": rpc error: code = NotFound desc = could not find container \"45a089731d0295032b19f898585280873b36d31f435f0e70ca14d8b1b5dfe97a\": container with ID starting with 45a089731d0295032b19f898585280873b36d31f435f0e70ca14d8b1b5dfe97a not found: ID does not exist" Sep 08 10:26:12.221771 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:12.221751 2576 scope.go:117] "RemoveContainer" containerID="c1a9612a9e578021f7df393bcd1b4b5c56cea81afc4adfc34579b847e611d94a" Sep 08 10:26:12.222100 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:26:12.222070 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a9612a9e578021f7df393bcd1b4b5c56cea81afc4adfc34579b847e611d94a\": container with ID starting with c1a9612a9e578021f7df393bcd1b4b5c56cea81afc4adfc34579b847e611d94a not found: ID does not exist" containerID="c1a9612a9e578021f7df393bcd1b4b5c56cea81afc4adfc34579b847e611d94a" Sep 08 10:26:12.222206 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:12.222118 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a9612a9e578021f7df393bcd1b4b5c56cea81afc4adfc34579b847e611d94a"} err="failed to get container status \"c1a9612a9e578021f7df393bcd1b4b5c56cea81afc4adfc34579b847e611d94a\": rpc error: code = NotFound desc = could not find container \"c1a9612a9e578021f7df393bcd1b4b5c56cea81afc4adfc34579b847e611d94a\": container with ID starting with c1a9612a9e578021f7df393bcd1b4b5c56cea81afc4adfc34579b847e611d94a not found: ID does not exist" Sep 08 10:26:12.225252 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:12.225229 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz"] Sep 08 10:26:12.229535 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:12.229512 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-af58b2-openshift-default-795b5bc54b-hjcvz"] Sep 08 10:26:12.679116 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:12.679083 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43e3c867-ae25-4228-b754-3904bf18b88b" path="/var/lib/kubelet/pods/43e3c867-ae25-4228-b754-3904bf18b88b/volumes" Sep 08 10:26:16.469567 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:16.469531 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:17.455543 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.455515 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb"] Sep 08 10:26:17.455783 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.455759 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" podUID="8d2ecc57-e0a2-42c4-b1af-74a678d967c9" containerName="maas-api" containerID="cri-o://8bea2c4404cd30916d2964b90e48d0d33c961974f9cec92571390d6dc26c5a65" gracePeriod=30 Sep 08 10:26:17.492544 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.492508 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:17.587628 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.587578 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:17.721393 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.721329 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:26:17.776760 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.776729 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-openshift-service-ca\") pod \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " Sep 08 10:26:17.776949 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.776771 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-metrics-certs\") pod \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " Sep 08 10:26:17.776949 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.776857 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7s2h\" (UniqueName: \"kubernetes.io/projected/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-kube-api-access-w7s2h\") pod \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " Sep 08 10:26:17.776949 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.776902 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-maas-api-tls\") pod \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\" (UID: \"8d2ecc57-e0a2-42c4-b1af-74a678d967c9\") " Sep 08 10:26:17.777158 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.777135 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8d2ecc57-e0a2-42c4-b1af-74a678d967c9" (UID: "8d2ecc57-e0a2-42c4-b1af-74a678d967c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:26:17.777305 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.777283 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-openshift-service-ca\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:26:17.779007 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.778977 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8d2ecc57-e0a2-42c4-b1af-74a678d967c9" (UID: "8d2ecc57-e0a2-42c4-b1af-74a678d967c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:26:17.779124 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.779034 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8d2ecc57-e0a2-42c4-b1af-74a678d967c9" (UID: "8d2ecc57-e0a2-42c4-b1af-74a678d967c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:26:17.779124 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.779041 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-kube-api-access-w7s2h" (OuterVolumeSpecName: "kube-api-access-w7s2h") pod "8d2ecc57-e0a2-42c4-b1af-74a678d967c9" (UID: "8d2ecc57-e0a2-42c4-b1af-74a678d967c9"). InnerVolumeSpecName "kube-api-access-w7s2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:26:17.878074 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.878038 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7s2h\" (UniqueName: \"kubernetes.io/projected/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-kube-api-access-w7s2h\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:26:17.878074 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.878072 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-maas-api-tls\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:26:17.878074 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:17.878081 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2ecc57-e0a2-42c4-b1af-74a678d967c9-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:26:18.241642 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:18.241577 2576 generic.go:356] "Generic (PLEG): container finished" podID="8d2ecc57-e0a2-42c4-b1af-74a678d967c9" containerID="8bea2c4404cd30916d2964b90e48d0d33c961974f9cec92571390d6dc26c5a65" exitCode=0 Sep 08 10:26:18.241823 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:18.241655 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" Sep 08 10:26:18.241823 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:18.241672 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" event={"ID":"8d2ecc57-e0a2-42c4-b1af-74a678d967c9","Type":"ContainerDied","Data":"8bea2c4404cd30916d2964b90e48d0d33c961974f9cec92571390d6dc26c5a65"} Sep 08 10:26:18.241823 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:18.241733 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb" event={"ID":"8d2ecc57-e0a2-42c4-b1af-74a678d967c9","Type":"ContainerDied","Data":"7afa1ff59ee15188ca40d11f800ffe60a052410e5f1ed5992e9954dc72d87930"} Sep 08 10:26:18.241823 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:18.241757 2576 scope.go:117] "RemoveContainer" containerID="8bea2c4404cd30916d2964b90e48d0d33c961974f9cec92571390d6dc26c5a65" Sep 08 10:26:18.255599 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:18.255578 2576 scope.go:117] "RemoveContainer" containerID="8bea2c4404cd30916d2964b90e48d0d33c961974f9cec92571390d6dc26c5a65" Sep 08 10:26:18.255991 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:26:18.255962 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bea2c4404cd30916d2964b90e48d0d33c961974f9cec92571390d6dc26c5a65\": container with ID starting with 8bea2c4404cd30916d2964b90e48d0d33c961974f9cec92571390d6dc26c5a65 not found: ID does not exist" containerID="8bea2c4404cd30916d2964b90e48d0d33c961974f9cec92571390d6dc26c5a65" Sep 08 10:26:18.256108 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:18.256001 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bea2c4404cd30916d2964b90e48d0d33c961974f9cec92571390d6dc26c5a65"} err="failed to get container status \"8bea2c4404cd30916d2964b90e48d0d33c961974f9cec92571390d6dc26c5a65\": rpc error: code = NotFound desc = could not find container \"8bea2c4404cd30916d2964b90e48d0d33c961974f9cec92571390d6dc26c5a65\": container with ID starting with 8bea2c4404cd30916d2964b90e48d0d33c961974f9cec92571390d6dc26c5a65 not found: ID does not exist" Sep 08 10:26:18.264084 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:18.264052 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb"] Sep 08 10:26:18.266771 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:18.266747 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ed05-78bb77b747-lkrjb"] Sep 08 10:26:18.371837 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:18.371786 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:18.440120 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:18.440090 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:18.678183 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:18.678138 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d2ecc57-e0a2-42c4-b1af-74a678d967c9" path="/var/lib/kubelet/pods/8d2ecc57-e0a2-42c4-b1af-74a678d967c9/volumes" Sep 08 10:26:22.477401 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:22.477369 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:22.552086 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:22.552049 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:26:25.337537 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:25.337503 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:25.395423 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:25.395387 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:31.215577 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:31.215545 2576 scope.go:117] "RemoveContainer" containerID="516cf5afa684fa045e09f65d3419ce5091181e8b2f6bb72ce3670eb1b8092a2c" Sep 08 10:26:42.156834 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:42.156795 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:26:42.157990 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:42.157964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:42.261231 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:42.261197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:53.698207 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:53.698120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:26:56.750796 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:26:56.750758 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:27:02.765978 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:02.765949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:03.758312 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:03.758268 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:04.224882 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.224845 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:04.495900 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.495822 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m"] Sep 08 10:27:04.496302 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496288 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99978507-8784-45cd-bd00-68f154580d0c" containerName="istio-proxy" Sep 08 10:27:04.496352 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496304 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="99978507-8784-45cd-bd00-68f154580d0c" containerName="istio-proxy" Sep 08 10:27:04.496352 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496323 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e3c867-ae25-4228-b754-3904bf18b88b" containerName="istio-proxy" Sep 08 10:27:04.496352 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496329 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e3c867-ae25-4228-b754-3904bf18b88b" containerName="istio-proxy" Sep 08 10:27:04.496352 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496335 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d2ecc57-e0a2-42c4-b1af-74a678d967c9" containerName="maas-api" Sep 08 10:27:04.496352 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496341 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d2ecc57-e0a2-42c4-b1af-74a678d967c9" containerName="maas-api" Sep 08 10:27:04.496556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496356 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1487d6-5336-417f-abe2-02788e8c2253" containerName="mt-curl-7890-627e92" Sep 08 10:27:04.496556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496362 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1487d6-5336-417f-abe2-02788e8c2253" containerName="mt-curl-7890-627e92" Sep 08 10:27:04.496556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496371 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a4c803-536a-452c-b61a-4dd51d0b2831" containerName="maas-api" Sep 08 10:27:04.496556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496376 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a4c803-536a-452c-b61a-4dd51d0b2831" containerName="maas-api" Sep 08 10:27:04.496556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496437 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb1f1345-2cdf-42b0-81ef-6a92a75a9c10" containerName="istio-proxy" Sep 08 10:27:04.496556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496443 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="99978507-8784-45cd-bd00-68f154580d0c" containerName="istio-proxy" Sep 08 10:27:04.496556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496449 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a4c803-536a-452c-b61a-4dd51d0b2831" containerName="maas-api" Sep 08 10:27:04.496556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496454 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e1487d6-5336-417f-abe2-02788e8c2253" containerName="mt-curl-7890-627e92" Sep 08 10:27:04.496556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496459 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d2ecc57-e0a2-42c4-b1af-74a678d967c9" containerName="maas-api" Sep 08 10:27:04.496556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496465 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e3c867-ae25-4228-b754-3904bf18b88b" containerName="istio-proxy" Sep 08 10:27:04.496556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.496471 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e3c867-ae25-4228-b754-3904bf18b88b" containerName="istio-proxy" Sep 08 10:27:04.499822 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.499800 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.502964 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.502942 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a0d0ba-openshift-default-dockercfg-g54lv\"" Sep 08 10:27:04.515260 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.515231 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m"] Sep 08 10:27:04.524591 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.524539 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:04.633741 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.633701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-istio-data\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.633741 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.633748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308881df-e633-4279-913c-6d19d1aa9274-istio-token\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.633980 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.633771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308881df-e633-4279-913c-6d19d1aa9274-istiod-ca-cert\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.633980 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.633853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308881df-e633-4279-913c-6d19d1aa9274-istio-podinfo\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.633980 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.633900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-workload-socket\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.633980 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.633939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-istio-envoy\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.634157 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.633986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-workload-certs\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.634157 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.634051 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvwrg\" (UniqueName: \"kubernetes.io/projected/308881df-e633-4279-913c-6d19d1aa9274-kube-api-access-rvwrg\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.634157 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.634092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-credential-socket\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.735563 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.735521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwrg\" (UniqueName: \"kubernetes.io/projected/308881df-e633-4279-913c-6d19d1aa9274-kube-api-access-rvwrg\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.735787 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.735575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-credential-socket\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.735787 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.735604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-istio-data\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.735787 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.735667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308881df-e633-4279-913c-6d19d1aa9274-istio-token\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.735787 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.735694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308881df-e633-4279-913c-6d19d1aa9274-istiod-ca-cert\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.735787 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.735716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308881df-e633-4279-913c-6d19d1aa9274-istio-podinfo\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.735787 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.735745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-workload-socket\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.736191 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.735876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-istio-envoy\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.736191 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.735945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-workload-certs\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.736191 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.736073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-istio-data\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.736191 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.736125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-credential-socket\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.736191 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.736185 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-workload-socket\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.736470 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.736356 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-workload-certs\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.736529 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.736510 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308881df-e633-4279-913c-6d19d1aa9274-istiod-ca-cert\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.738134 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.738106 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-istio-envoy\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.738345 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.738328 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308881df-e633-4279-913c-6d19d1aa9274-istio-podinfo\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.743142 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.743117 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308881df-e633-4279-913c-6d19d1aa9274-istio-token\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.743239 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.743169 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwrg\" (UniqueName: \"kubernetes.io/projected/308881df-e633-4279-913c-6d19d1aa9274-kube-api-access-rvwrg\") pod \"e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:04.815677 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:04.815567 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:05.098878 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:05.098843 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m"] Sep 08 10:27:05.100051 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:27:05.100024 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308881df_e633_4279_913c_6d19d1aa9274.slice/crio-b427c789920ffa2e5e160e39e3c60b1bd98b7ad06c0e240cc78537bca68380bc WatchSource:0}: Error finding container b427c789920ffa2e5e160e39e3c60b1bd98b7ad06c0e240cc78537bca68380bc: Status 404 returned error can't find the container with id b427c789920ffa2e5e160e39e3c60b1bd98b7ad06c0e240cc78537bca68380bc Sep 08 10:27:05.102363 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:05.102330 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:27:05.102431 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:05.102400 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:27:05.102469 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:05.102430 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:27:05.526669 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:05.526624 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" event={"ID":"308881df-e633-4279-913c-6d19d1aa9274","Type":"ContainerStarted","Data":"9f2a67139a0f94aa785703528997905cd1febf33015c740ae5626754d61a33ba"} Sep 08 10:27:05.527146 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:05.526696 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" event={"ID":"308881df-e633-4279-913c-6d19d1aa9274","Type":"ContainerStarted","Data":"b427c789920ffa2e5e160e39e3c60b1bd98b7ad06c0e240cc78537bca68380bc"} Sep 08 10:27:05.551282 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:05.551234 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" podStartSLOduration=1.551219941 podStartE2EDuration="1.551219941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:27:05.546007052 +0000 UTC m=+1535.335058857" watchObservedRunningTime="2026-09-08 10:27:05.551219941 +0000 UTC m=+1535.340271745" Sep 08 10:27:05.816483 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:05.816397 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:06.209338 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:06.209289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:06.821729 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:06.821700 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:07.478692 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:07.478663 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:07.537541 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:07.537510 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:07.538472 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:07.538455 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:07.596645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:07.596575 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:08.011651 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:08.011600 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:27:08.623394 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:08.623361 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:08.724602 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:08.724568 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:08.880769 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:08.880690 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:09.225906 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:09.225876 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:09.327063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:09.327029 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:19.152528 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:19.152480 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:27:23.206496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:23.205084 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": read tcp 10.134.0.2:50164->10.134.0.82:15021: read: connection reset by peer" start-of-body= Sep 08 10:27:23.206496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:23.205152 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" podUID="308881df-e633-4279-913c-6d19d1aa9274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": read tcp 10.134.0.2:50164->10.134.0.82:15021: read: connection reset by peer" Sep 08 10:27:23.649587 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:23.649555 2576 generic.go:356] "Generic (PLEG): container finished" podID="308881df-e633-4279-913c-6d19d1aa9274" containerID="9f2a67139a0f94aa785703528997905cd1febf33015c740ae5626754d61a33ba" exitCode=0 Sep 08 10:27:23.649799 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:23.649646 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" event={"ID":"308881df-e633-4279-913c-6d19d1aa9274","Type":"ContainerDied","Data":"9f2a67139a0f94aa785703528997905cd1febf33015c740ae5626754d61a33ba"} Sep 08 10:27:23.650090 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:23.650076 2576 scope.go:117] "RemoveContainer" containerID="9f2a67139a0f94aa785703528997905cd1febf33015c740ae5626754d61a33ba" Sep 08 10:27:23.651218 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:23.651192 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:27:23.651279 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:23.651262 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:27:23.651315 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:23.651293 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:27:24.658599 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:24.658561 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" event={"ID":"308881df-e633-4279-913c-6d19d1aa9274","Type":"ContainerStarted","Data":"ca55855f906c16b4e1cb20bd26cdafbdc59495dfaf69862cbcfdd0e280070edf"} Sep 08 10:27:24.816381 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:24.816330 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:24.816611 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:24.816547 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 08 10:27:24.816611 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:24.816602 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" podUID="308881df-e633-4279-913c-6d19d1aa9274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 08 10:27:25.816520 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:25.816478 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 08 10:27:25.816953 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:25.816550 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" podUID="308881df-e633-4279-913c-6d19d1aa9274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 08 10:27:26.816032 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:26.815980 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 08 10:27:26.816229 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:26.816071 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" podUID="308881df-e633-4279-913c-6d19d1aa9274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 08 10:27:27.816795 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:27.816758 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 08 10:27:27.817232 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:27.816830 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" podUID="308881df-e633-4279-913c-6d19d1aa9274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 08 10:27:28.816319 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:28.816278 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 08 10:27:28.816492 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:28.816348 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" podUID="308881df-e633-4279-913c-6d19d1aa9274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 08 10:27:29.861183 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:29.861150 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:29.861674 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:29.861473 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:29.862342 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:29.862323 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:27:31.262305 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:31.262271 2576 scope.go:117] "RemoveContainer" containerID="1727c0af97e890f45cc36fc4624ca1f95942dd4736c5515898b97f3a84f8f4ab" Sep 08 10:27:32.486476 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:32.486440 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:33.177924 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.177890 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:33.458411 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.458334 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d"] Sep 08 10:27:33.458980 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.458964 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e3c867-ae25-4228-b754-3904bf18b88b" containerName="istio-proxy" Sep 08 10:27:33.459032 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.458984 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e3c867-ae25-4228-b754-3904bf18b88b" containerName="istio-proxy" Sep 08 10:27:33.459032 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.459020 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99978507-8784-45cd-bd00-68f154580d0c" containerName="istio-proxy" Sep 08 10:27:33.459032 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.459029 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="99978507-8784-45cd-bd00-68f154580d0c" containerName="istio-proxy" Sep 08 10:27:33.459133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.459118 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="99978507-8784-45cd-bd00-68f154580d0c" containerName="istio-proxy" Sep 08 10:27:33.462610 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.462589 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.465138 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.465116 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-86902f-openshift-default-dockercfg-4n2gx\"" Sep 08 10:27:33.473709 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.473684 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d"] Sep 08 10:27:33.494564 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.494535 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:33.518527 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.518495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-istio-data\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.518733 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.518535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-workload-socket\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.518733 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.518656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-istio-envoy\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.518733 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.518706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-workload-certs\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.518733 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.518725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfd449a-27df-478f-ae15-b3e35398f134-istio-token\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.518930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.518763 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-credential-socket\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.518930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.518802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dfd449a-27df-478f-ae15-b3e35398f134-istio-podinfo\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.518930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.518818 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s555n\" (UniqueName: \"kubernetes.io/projected/9dfd449a-27df-478f-ae15-b3e35398f134-kube-api-access-s555n\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.518930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.518860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dfd449a-27df-478f-ae15-b3e35398f134-istiod-ca-cert\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.619708 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.619668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-istio-envoy\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.619914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.619751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-workload-certs\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.619914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.619773 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfd449a-27df-478f-ae15-b3e35398f134-istio-token\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.619914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.619802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-credential-socket\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.619914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.619836 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dfd449a-27df-478f-ae15-b3e35398f134-istio-podinfo\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.619914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.619858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s555n\" (UniqueName: \"kubernetes.io/projected/9dfd449a-27df-478f-ae15-b3e35398f134-kube-api-access-s555n\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.620188 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.619918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dfd449a-27df-478f-ae15-b3e35398f134-istiod-ca-cert\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.620188 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.619958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-istio-data\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.620188 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.619991 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-workload-socket\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.620361 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.620247 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-workload-certs\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.620361 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.620281 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-credential-socket\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.620467 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.620429 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-workload-socket\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.620630 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.620592 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-istio-data\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.620822 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.620802 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dfd449a-27df-478f-ae15-b3e35398f134-istiod-ca-cert\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.622195 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.622170 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-istio-envoy\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.622380 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.622362 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dfd449a-27df-478f-ae15-b3e35398f134-istio-podinfo\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.628103 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.628079 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfd449a-27df-478f-ae15-b3e35398f134-istio-token\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.628191 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.628176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s555n\" (UniqueName: \"kubernetes.io/projected/9dfd449a-27df-478f-ae15-b3e35398f134-kube-api-access-s555n\") pod \"e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:33.776350 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:33.776248 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:34.004263 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:34.004225 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:34.064001 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:34.063972 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d"] Sep 08 10:27:34.064687 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:27:34.064659 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dfd449a_27df_478f_ae15_b3e35398f134.slice/crio-480220e3d5d176dae6cca2a6f2227be7fd51129dee30bb1dbf6616fc477aeb0e WatchSource:0}: Error finding container 480220e3d5d176dae6cca2a6f2227be7fd51129dee30bb1dbf6616fc477aeb0e: Status 404 returned error can't find the container with id 480220e3d5d176dae6cca2a6f2227be7fd51129dee30bb1dbf6616fc477aeb0e Sep 08 10:27:34.067004 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:34.066970 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:27:34.067108 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:34.067045 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:27:34.067108 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:34.067074 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:27:34.718323 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:34.718281 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" event={"ID":"9dfd449a-27df-478f-ae15-b3e35398f134","Type":"ContainerStarted","Data":"787a52a8c12607f37d9f3e6b767844f3d40e54fc1f7227858463eee9fa02ef49"} Sep 08 10:27:34.718732 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:34.718350 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" event={"ID":"9dfd449a-27df-478f-ae15-b3e35398f134","Type":"ContainerStarted","Data":"480220e3d5d176dae6cca2a6f2227be7fd51129dee30bb1dbf6616fc477aeb0e"} Sep 08 10:27:34.739321 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:34.739266 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" podStartSLOduration=1.73925075 podStartE2EDuration="1.73925075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:27:34.737562656 +0000 UTC m=+1564.526614501" watchObservedRunningTime="2026-09-08 10:27:34.73925075 +0000 UTC m=+1564.528302555" Sep 08 10:27:34.776677 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:34.776644 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:35.782306 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:35.782274 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:36.713784 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:36.713747 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:36.730155 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:36.730124 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:36.731057 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:36.731039 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:37.802575 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:37.802536 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm"] Sep 08 10:27:37.807074 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:37.807049 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:37.809402 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:37.809382 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-86902f-serving-cert\"" Sep 08 10:27:37.820001 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:37.819966 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm"] Sep 08 10:27:37.866452 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:37.866412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b9899e-6753-4d09-af1c-8e6a9e6c4477-maas-api-tls\") pod \"maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:37.866715 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:37.866523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b9899e-6753-4d09-af1c-8e6a9e6c4477-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:37.866715 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:37.866553 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57b9899e-6753-4d09-af1c-8e6a9e6c4477-metrics-certs\") pod \"maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:37.866878 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:37.866710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-827sd\" (UniqueName: \"kubernetes.io/projected/57b9899e-6753-4d09-af1c-8e6a9e6c4477-kube-api-access-827sd\") pod \"maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:37.968216 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:37.968164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b9899e-6753-4d09-af1c-8e6a9e6c4477-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:37.968401 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:37.968225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57b9899e-6753-4d09-af1c-8e6a9e6c4477-metrics-certs\") pod \"maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:37.968401 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:37.968306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-827sd\" (UniqueName: \"kubernetes.io/projected/57b9899e-6753-4d09-af1c-8e6a9e6c4477-kube-api-access-827sd\") pod \"maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:37.968401 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:37.968332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b9899e-6753-4d09-af1c-8e6a9e6c4477-maas-api-tls\") pod \"maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:37.968518 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:27:37.968460 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-serving-cert: secret "maas-api-e2e-ipp-b-86902f-serving-cert" not found Sep 08 10:27:37.968554 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:27:37.968536 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57b9899e-6753-4d09-af1c-8e6a9e6c4477-maas-api-tls podName:57b9899e-6753-4d09-af1c-8e6a9e6c4477 nodeName:}" failed. No retries permitted until 2026-09-08 10:27:38.468514894 +0000 UTC m=+1568.257566680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57b9899e-6753-4d09-af1c-8e6a9e6c4477-maas-api-tls") pod "maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" (UID: "57b9899e-6753-4d09-af1c-8e6a9e6c4477") : secret "maas-api-e2e-ipp-b-86902f-serving-cert" not found Sep 08 10:27:37.968938 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:37.968916 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b9899e-6753-4d09-af1c-8e6a9e6c4477-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:37.970843 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:37.970820 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57b9899e-6753-4d09-af1c-8e6a9e6c4477-metrics-certs\") pod \"maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:37.980856 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:37.980831 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-827sd\" (UniqueName: \"kubernetes.io/projected/57b9899e-6753-4d09-af1c-8e6a9e6c4477-kube-api-access-827sd\") pod \"maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:38.246521 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:38.246490 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:38.360045 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:38.360005 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:38.465125 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:38.465090 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:38.474430 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:38.474389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b9899e-6753-4d09-af1c-8e6a9e6c4477-maas-api-tls\") pod \"maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:38.476993 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:38.476970 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b9899e-6753-4d09-af1c-8e6a9e6c4477-maas-api-tls\") pod \"maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:38.569797 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:38.569704 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:38.722857 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:38.722820 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:39.021181 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:39.021152 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm"] Sep 08 10:27:39.022753 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:27:39.022727 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b9899e_6753_4d09_af1c_8e6a9e6c4477.slice/crio-25c11d5e52b6bc43b98a9aa35e62e714d89e2dc4dcb5b914ea4d74ab3074037b WatchSource:0}: Error finding container 25c11d5e52b6bc43b98a9aa35e62e714d89e2dc4dcb5b914ea4d74ab3074037b: Status 404 returned error can't find the container with id 25c11d5e52b6bc43b98a9aa35e62e714d89e2dc4dcb5b914ea4d74ab3074037b Sep 08 10:27:39.750680 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:39.750551 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" event={"ID":"57b9899e-6753-4d09-af1c-8e6a9e6c4477","Type":"ContainerStarted","Data":"f7f96bfe2cfc4ded38c612e3ca157dbe4f01cc3f1a8639c63f711b3f66ceccc5"} Sep 08 10:27:39.750680 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:39.750613 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" event={"ID":"57b9899e-6753-4d09-af1c-8e6a9e6c4477","Type":"ContainerStarted","Data":"25c11d5e52b6bc43b98a9aa35e62e714d89e2dc4dcb5b914ea4d74ab3074037b"} Sep 08 10:27:39.750680 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:39.750678 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:39.769503 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:39.769446 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" podStartSLOduration=2.39123206 podStartE2EDuration="2.769428312s" totalImagesPullingTime="378.196252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:27:39.768532072 +0000 UTC m=+1569.557583877" watchObservedRunningTime="2026-09-08 10:27:39.769428312 +0000 UTC m=+1569.558480115" Sep 08 10:27:40.923359 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:40.923325 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:40.996726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:40.996654 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:27:44.555126 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:44.555092 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:27:45.761893 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:45.761862 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:27:55.864746 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:55.864709 2576 generic.go:356] "Generic (PLEG): container finished" podID="9dfd449a-27df-478f-ae15-b3e35398f134" containerID="787a52a8c12607f37d9f3e6b767844f3d40e54fc1f7227858463eee9fa02ef49" exitCode=0 Sep 08 10:27:55.865165 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:55.864796 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" event={"ID":"9dfd449a-27df-478f-ae15-b3e35398f134","Type":"ContainerDied","Data":"787a52a8c12607f37d9f3e6b767844f3d40e54fc1f7227858463eee9fa02ef49"} Sep 08 10:27:55.865408 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:55.865390 2576 scope.go:117] "RemoveContainer" containerID="787a52a8c12607f37d9f3e6b767844f3d40e54fc1f7227858463eee9fa02ef49" Sep 08 10:27:55.866756 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:55.866730 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:27:55.866840 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:55.866793 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:27:55.866840 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:55.866834 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 10:27:56.873210 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:56.873173 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" event={"ID":"9dfd449a-27df-478f-ae15-b3e35398f134","Type":"ContainerStarted","Data":"0c58360ff58ebb563b1979c0701fb883fe6fba79f1ba84bfcb369f64224b08e0"} Sep 08 10:27:57.776375 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:57.776339 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:27:57.776730 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:57.776712 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 08 10:27:57.776789 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:57.776755 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" podUID="9dfd449a-27df-478f-ae15-b3e35398f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 08 10:27:58.776881 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:58.776841 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 08 10:27:58.777284 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:58.776921 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" podUID="9dfd449a-27df-478f-ae15-b3e35398f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 08 10:27:59.776825 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:59.776772 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 08 10:27:59.777023 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:27:59.776858 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" podUID="9dfd449a-27df-478f-ae15-b3e35398f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 08 10:28:00.777330 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:00.777294 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 08 10:28:00.777733 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:00.777350 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" podUID="9dfd449a-27df-478f-ae15-b3e35398f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 08 10:28:01.758423 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:01.758388 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:28:01.776792 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:01.776759 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 08 10:28:01.776968 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:01.776826 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" podUID="9dfd449a-27df-478f-ae15-b3e35398f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 08 10:28:02.816415 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:02.816383 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:28:02.816914 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:02.816807 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:28:02.818059 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:02.818033 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:28:12.067945 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:12.067910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:28:23.888729 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:23.888639 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:28:28.459577 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:28.459535 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:28:49.653001 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:49.652967 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:28:55.488569 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:55.488533 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:28:55.794832 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:55.794745 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:28:56.181802 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:56.181752 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:28:58.565417 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:58.565386 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:28:58.711995 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:58.711965 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:28:59.778632 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.778588 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z"] Sep 08 10:28:59.783267 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.783238 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:28:59.785685 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.785662 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-56bb3c-serving-cert\"" Sep 08 10:28:59.794799 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.794767 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z"] Sep 08 10:28:59.814082 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.814058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:28:59.891288 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.891250 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23675580-e81a-4cc3-9539-9e109879d0c9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:28:59.891498 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.891317 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23675580-e81a-4cc3-9539-9e109879d0c9-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:28:59.891498 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.891358 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9tl4\" (UniqueName: \"kubernetes.io/projected/23675580-e81a-4cc3-9539-9e109879d0c9-kube-api-access-x9tl4\") pod \"maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:28:59.891498 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.891389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23675580-e81a-4cc3-9539-9e109879d0c9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:28:59.915279 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.915248 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:28:59.992495 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.992470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23675580-e81a-4cc3-9539-9e109879d0c9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:28:59.992672 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.992510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23675580-e81a-4cc3-9539-9e109879d0c9-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:28:59.992672 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.992546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9tl4\" (UniqueName: \"kubernetes.io/projected/23675580-e81a-4cc3-9539-9e109879d0c9-kube-api-access-x9tl4\") pod \"maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:28:59.992672 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.992579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23675580-e81a-4cc3-9539-9e109879d0c9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:28:59.992886 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:28:59.992833 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-serving-cert: secret "maas-api-e2e-ipp-cleanup-56bb3c-serving-cert" not found Sep 08 10:28:59.993000 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:28:59.992975 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23675580-e81a-4cc3-9539-9e109879d0c9-maas-api-tls podName:23675580-e81a-4cc3-9539-9e109879d0c9 nodeName:}" failed. No retries permitted until 2026-09-08 10:29:00.492950261 +0000 UTC m=+1650.282002058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23675580-e81a-4cc3-9539-9e109879d0c9-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" (UID: "23675580-e81a-4cc3-9539-9e109879d0c9") : secret "maas-api-e2e-ipp-cleanup-56bb3c-serving-cert" not found Sep 08 10:28:59.993362 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.993341 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23675580-e81a-4cc3-9539-9e109879d0c9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:28:59.995041 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:28:59.995022 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23675580-e81a-4cc3-9539-9e109879d0c9-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:29:00.003349 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:00.003324 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9tl4\" (UniqueName: \"kubernetes.io/projected/23675580-e81a-4cc3-9539-9e109879d0c9-kube-api-access-x9tl4\") pod \"maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:29:00.364083 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:00.364057 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:00.462677 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:00.462649 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:00.498297 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:00.498259 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23675580-e81a-4cc3-9539-9e109879d0c9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:29:00.500913 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:00.500876 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23675580-e81a-4cc3-9539-9e109879d0c9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:29:00.696457 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:00.696423 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:29:00.993822 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:00.993793 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z"] Sep 08 10:29:00.994661 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:29:00.994604 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23675580_e81a_4cc3_9539_9e109879d0c9.slice/crio-050a42c2ecc074fb27cfbea2ac4fab01dc1f049336c549d9ed822009d961efe7 WatchSource:0}: Error finding container 050a42c2ecc074fb27cfbea2ac4fab01dc1f049336c549d9ed822009d961efe7: Status 404 returned error can't find the container with id 050a42c2ecc074fb27cfbea2ac4fab01dc1f049336c549d9ed822009d961efe7 Sep 08 10:29:01.271897 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:01.271803 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" event={"ID":"23675580-e81a-4cc3-9539-9e109879d0c9","Type":"ContainerStarted","Data":"050a42c2ecc074fb27cfbea2ac4fab01dc1f049336c549d9ed822009d961efe7"} Sep 08 10:29:02.279546 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:02.279510 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" event={"ID":"23675580-e81a-4cc3-9539-9e109879d0c9","Type":"ContainerStarted","Data":"a7a1d3d02f3c4552ae5ce46a1436b84ac67e74909a4a0caabd8029281e2e6886"} Sep 08 10:29:02.279997 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:02.279640 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:29:02.304320 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:02.304242 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" podStartSLOduration=2.842629933 podStartE2EDuration="3.304218808s" totalImagesPullingTime="461.588875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:28:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:29:02.302141794 +0000 UTC m=+1652.091193599" watchObservedRunningTime="2026-09-08 10:29:02.304218808 +0000 UTC m=+1652.093270611" Sep 08 10:29:08.255931 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:08.255889 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:29:08.290587 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:08.290559 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:29:23.482754 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:23.482718 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:30.192487 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.192454 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z"] Sep 08 10:29:30.192981 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.192725 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" podUID="23675580-e81a-4cc3-9539-9e109879d0c9" containerName="maas-api" containerID="cri-o://a7a1d3d02f3c4552ae5ce46a1436b84ac67e74909a4a0caabd8029281e2e6886" gracePeriod=30 Sep 08 10:29:30.306997 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.306964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:30.408308 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.408274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:30.443277 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.443217 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:29:30.459935 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.459912 2576 generic.go:356] "Generic (PLEG): container finished" podID="23675580-e81a-4cc3-9539-9e109879d0c9" containerID="a7a1d3d02f3c4552ae5ce46a1436b84ac67e74909a4a0caabd8029281e2e6886" exitCode=0 Sep 08 10:29:30.460054 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.459946 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" Sep 08 10:29:30.460054 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.459962 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" event={"ID":"23675580-e81a-4cc3-9539-9e109879d0c9","Type":"ContainerDied","Data":"a7a1d3d02f3c4552ae5ce46a1436b84ac67e74909a4a0caabd8029281e2e6886"} Sep 08 10:29:30.460054 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.460013 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z" event={"ID":"23675580-e81a-4cc3-9539-9e109879d0c9","Type":"ContainerDied","Data":"050a42c2ecc074fb27cfbea2ac4fab01dc1f049336c549d9ed822009d961efe7"} Sep 08 10:29:30.460054 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.460036 2576 scope.go:117] "RemoveContainer" containerID="a7a1d3d02f3c4552ae5ce46a1436b84ac67e74909a4a0caabd8029281e2e6886" Sep 08 10:29:30.468740 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.468692 2576 scope.go:117] "RemoveContainer" containerID="a7a1d3d02f3c4552ae5ce46a1436b84ac67e74909a4a0caabd8029281e2e6886" Sep 08 10:29:30.469005 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:29:30.468984 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a1d3d02f3c4552ae5ce46a1436b84ac67e74909a4a0caabd8029281e2e6886\": container with ID starting with a7a1d3d02f3c4552ae5ce46a1436b84ac67e74909a4a0caabd8029281e2e6886 not found: ID does not exist" containerID="a7a1d3d02f3c4552ae5ce46a1436b84ac67e74909a4a0caabd8029281e2e6886" Sep 08 10:29:30.469076 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.469017 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a1d3d02f3c4552ae5ce46a1436b84ac67e74909a4a0caabd8029281e2e6886"} err="failed to get container status \"a7a1d3d02f3c4552ae5ce46a1436b84ac67e74909a4a0caabd8029281e2e6886\": rpc error: code = NotFound desc = could not find container \"a7a1d3d02f3c4552ae5ce46a1436b84ac67e74909a4a0caabd8029281e2e6886\": container with ID starting with a7a1d3d02f3c4552ae5ce46a1436b84ac67e74909a4a0caabd8029281e2e6886 not found: ID does not exist" Sep 08 10:29:30.507182 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.507158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:30.605638 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.601893 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9tl4\" (UniqueName: \"kubernetes.io/projected/23675580-e81a-4cc3-9539-9e109879d0c9-kube-api-access-x9tl4\") pod \"23675580-e81a-4cc3-9539-9e109879d0c9\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " Sep 08 10:29:30.605638 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.601985 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23675580-e81a-4cc3-9539-9e109879d0c9-maas-api-tls\") pod \"23675580-e81a-4cc3-9539-9e109879d0c9\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " Sep 08 10:29:30.605638 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.602086 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23675580-e81a-4cc3-9539-9e109879d0c9-metrics-certs\") pod \"23675580-e81a-4cc3-9539-9e109879d0c9\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " Sep 08 10:29:30.605638 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.602130 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23675580-e81a-4cc3-9539-9e109879d0c9-openshift-service-ca\") pod \"23675580-e81a-4cc3-9539-9e109879d0c9\" (UID: \"23675580-e81a-4cc3-9539-9e109879d0c9\") " Sep 08 10:29:30.605638 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.602872 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23675580-e81a-4cc3-9539-9e109879d0c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "23675580-e81a-4cc3-9539-9e109879d0c9" (UID: "23675580-e81a-4cc3-9539-9e109879d0c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:29:30.609637 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.606313 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23675580-e81a-4cc3-9539-9e109879d0c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "23675580-e81a-4cc3-9539-9e109879d0c9" (UID: "23675580-e81a-4cc3-9539-9e109879d0c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:29:30.611881 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.610772 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23675580-e81a-4cc3-9539-9e109879d0c9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "23675580-e81a-4cc3-9539-9e109879d0c9" (UID: "23675580-e81a-4cc3-9539-9e109879d0c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:29:30.611881 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.610796 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23675580-e81a-4cc3-9539-9e109879d0c9-kube-api-access-x9tl4" (OuterVolumeSpecName: "kube-api-access-x9tl4") pod "23675580-e81a-4cc3-9539-9e109879d0c9" (UID: "23675580-e81a-4cc3-9539-9e109879d0c9"). InnerVolumeSpecName "kube-api-access-x9tl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:29:30.703686 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.703612 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9tl4\" (UniqueName: \"kubernetes.io/projected/23675580-e81a-4cc3-9539-9e109879d0c9-kube-api-access-x9tl4\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:29:30.703686 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.703650 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23675580-e81a-4cc3-9539-9e109879d0c9-maas-api-tls\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:29:30.703686 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.703658 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23675580-e81a-4cc3-9539-9e109879d0c9-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:29:30.703686 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.703666 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23675580-e81a-4cc3-9539-9e109879d0c9-openshift-service-ca\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:29:30.778465 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.778435 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z"] Sep 08 10:29:30.781001 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.780981 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-56bb3c-58dd748546-rrs8z"] Sep 08 10:29:30.828381 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:30.828354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:31.293407 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:31.293368 2576 scope.go:117] "RemoveContainer" containerID="e976eb3f24810edfcbc93b7d2b525b4f86de0f7ece251806f78cd57312789f52" Sep 08 10:29:32.677758 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:32.677726 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23675580-e81a-4cc3-9539-9e109879d0c9" path="/var/lib/kubelet/pods/23675580-e81a-4cc3-9539-9e109879d0c9/volumes" Sep 08 10:29:36.148351 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:36.148307 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:29:36.646238 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:36.646202 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:36.749825 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:36.749793 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:36.935061 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:36.934959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:39.086913 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:39.086852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:40.791161 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:40.791129 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:56.159126 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:56.159037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:56.258907 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:56.258862 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:56.358787 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:56.358743 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:56.903611 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:56.903580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:29:56.954240 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:56.954207 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:29:56.963666 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:29:56.963639 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:30:01.946105 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:01.946066 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:30:06.552835 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:06.552791 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:30:07.548217 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:07.548171 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:30:38.583407 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:38.583371 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:30:39.022058 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:39.022025 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:30:44.630286 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:44.630254 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:30:44.699042 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:44.699003 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:30:44.898674 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:44.898634 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m"] Sep 08 10:30:44.898912 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:44.898887 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" podUID="308881df-e633-4279-913c-6d19d1aa9274" containerName="istio-proxy" containerID="cri-o://ca55855f906c16b4e1cb20bd26cdafbdc59495dfaf69862cbcfdd0e280070edf" gracePeriod=30 Sep 08 10:30:44.926049 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:44.926020 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:30:45.438397 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:45.438368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:30:50.158507 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.158449 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:30:50.290375 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.290338 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-credential-socket\") pod \"308881df-e633-4279-913c-6d19d1aa9274\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " Sep 08 10:30:50.290549 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.290383 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308881df-e633-4279-913c-6d19d1aa9274-istio-podinfo\") pod \"308881df-e633-4279-913c-6d19d1aa9274\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " Sep 08 10:30:50.290549 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.290441 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-istio-envoy\") pod \"308881df-e633-4279-913c-6d19d1aa9274\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " Sep 08 10:30:50.290549 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.290475 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-workload-socket\") pod \"308881df-e633-4279-913c-6d19d1aa9274\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " Sep 08 10:30:50.290549 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.290507 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-istio-data\") pod \"308881df-e633-4279-913c-6d19d1aa9274\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " Sep 08 10:30:50.290866 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.290570 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308881df-e633-4279-913c-6d19d1aa9274-istiod-ca-cert\") pod \"308881df-e633-4279-913c-6d19d1aa9274\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " Sep 08 10:30:50.290866 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.290636 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvwrg\" (UniqueName: \"kubernetes.io/projected/308881df-e633-4279-913c-6d19d1aa9274-kube-api-access-rvwrg\") pod \"308881df-e633-4279-913c-6d19d1aa9274\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " Sep 08 10:30:50.290866 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.290671 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-workload-certs\") pod \"308881df-e633-4279-913c-6d19d1aa9274\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " Sep 08 10:30:50.290866 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.290673 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "308881df-e633-4279-913c-6d19d1aa9274" (UID: "308881df-e633-4279-913c-6d19d1aa9274"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:30:50.290866 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.290697 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308881df-e633-4279-913c-6d19d1aa9274-istio-token\") pod \"308881df-e633-4279-913c-6d19d1aa9274\" (UID: \"308881df-e633-4279-913c-6d19d1aa9274\") " Sep 08 10:30:50.290866 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.290832 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-istio-data" (OuterVolumeSpecName: "istio-data") pod "308881df-e633-4279-913c-6d19d1aa9274" (UID: "308881df-e633-4279-913c-6d19d1aa9274"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:30:50.291185 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.291115 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-credential-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:30:50.291185 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.291125 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "308881df-e633-4279-913c-6d19d1aa9274" (UID: "308881df-e633-4279-913c-6d19d1aa9274"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:30:50.291185 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.291138 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-istio-data\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:30:50.291345 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.291300 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "308881df-e633-4279-913c-6d19d1aa9274" (UID: "308881df-e633-4279-913c-6d19d1aa9274"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:30:50.291345 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.291307 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308881df-e633-4279-913c-6d19d1aa9274-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "308881df-e633-4279-913c-6d19d1aa9274" (UID: "308881df-e633-4279-913c-6d19d1aa9274"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:30:50.292863 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.292829 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/308881df-e633-4279-913c-6d19d1aa9274-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "308881df-e633-4279-913c-6d19d1aa9274" (UID: "308881df-e633-4279-913c-6d19d1aa9274"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 10:30:50.292994 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.292929 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "308881df-e633-4279-913c-6d19d1aa9274" (UID: "308881df-e633-4279-913c-6d19d1aa9274"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:30:50.293216 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.293192 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308881df-e633-4279-913c-6d19d1aa9274-istio-token" (OuterVolumeSpecName: "istio-token") pod "308881df-e633-4279-913c-6d19d1aa9274" (UID: "308881df-e633-4279-913c-6d19d1aa9274"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:30:50.293216 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.293188 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308881df-e633-4279-913c-6d19d1aa9274-kube-api-access-rvwrg" (OuterVolumeSpecName: "kube-api-access-rvwrg") pod "308881df-e633-4279-913c-6d19d1aa9274" (UID: "308881df-e633-4279-913c-6d19d1aa9274"). InnerVolumeSpecName "kube-api-access-rvwrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:30:50.391894 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.391871 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308881df-e633-4279-913c-6d19d1aa9274-istiod-ca-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:30:50.391894 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.391891 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvwrg\" (UniqueName: \"kubernetes.io/projected/308881df-e633-4279-913c-6d19d1aa9274-kube-api-access-rvwrg\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:30:50.392036 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.391902 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-workload-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:30:50.392036 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.391912 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308881df-e633-4279-913c-6d19d1aa9274-istio-token\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:30:50.392036 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.391920 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308881df-e633-4279-913c-6d19d1aa9274-istio-podinfo\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:30:50.392036 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.391927 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-istio-envoy\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:30:50.392036 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.391934 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308881df-e633-4279-913c-6d19d1aa9274-workload-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:30:50.931019 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.930986 2576 generic.go:356] "Generic (PLEG): container finished" podID="308881df-e633-4279-913c-6d19d1aa9274" containerID="ca55855f906c16b4e1cb20bd26cdafbdc59495dfaf69862cbcfdd0e280070edf" exitCode=0 Sep 08 10:30:50.931188 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.931039 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" Sep 08 10:30:50.931188 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.931068 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" event={"ID":"308881df-e633-4279-913c-6d19d1aa9274","Type":"ContainerDied","Data":"ca55855f906c16b4e1cb20bd26cdafbdc59495dfaf69862cbcfdd0e280070edf"} Sep 08 10:30:50.931188 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.931124 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m" event={"ID":"308881df-e633-4279-913c-6d19d1aa9274","Type":"ContainerDied","Data":"b427c789920ffa2e5e160e39e3c60b1bd98b7ad06c0e240cc78537bca68380bc"} Sep 08 10:30:50.931188 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.931146 2576 scope.go:117] "RemoveContainer" containerID="ca55855f906c16b4e1cb20bd26cdafbdc59495dfaf69862cbcfdd0e280070edf" Sep 08 10:30:50.940364 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.940347 2576 scope.go:117] "RemoveContainer" containerID="9f2a67139a0f94aa785703528997905cd1febf33015c740ae5626754d61a33ba" Sep 08 10:30:50.948317 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.948301 2576 scope.go:117] "RemoveContainer" containerID="ca55855f906c16b4e1cb20bd26cdafbdc59495dfaf69862cbcfdd0e280070edf" Sep 08 10:30:50.948557 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:30:50.948540 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca55855f906c16b4e1cb20bd26cdafbdc59495dfaf69862cbcfdd0e280070edf\": container with ID starting with ca55855f906c16b4e1cb20bd26cdafbdc59495dfaf69862cbcfdd0e280070edf not found: ID does not exist" containerID="ca55855f906c16b4e1cb20bd26cdafbdc59495dfaf69862cbcfdd0e280070edf" Sep 08 10:30:50.948603 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.948565 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca55855f906c16b4e1cb20bd26cdafbdc59495dfaf69862cbcfdd0e280070edf"} err="failed to get container status \"ca55855f906c16b4e1cb20bd26cdafbdc59495dfaf69862cbcfdd0e280070edf\": rpc error: code = NotFound desc = could not find container \"ca55855f906c16b4e1cb20bd26cdafbdc59495dfaf69862cbcfdd0e280070edf\": container with ID starting with ca55855f906c16b4e1cb20bd26cdafbdc59495dfaf69862cbcfdd0e280070edf not found: ID does not exist" Sep 08 10:30:50.948603 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.948581 2576 scope.go:117] "RemoveContainer" containerID="9f2a67139a0f94aa785703528997905cd1febf33015c740ae5626754d61a33ba" Sep 08 10:30:50.948827 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:30:50.948809 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2a67139a0f94aa785703528997905cd1febf33015c740ae5626754d61a33ba\": container with ID starting with 9f2a67139a0f94aa785703528997905cd1febf33015c740ae5626754d61a33ba not found: ID does not exist" containerID="9f2a67139a0f94aa785703528997905cd1febf33015c740ae5626754d61a33ba" Sep 08 10:30:50.948886 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.948833 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2a67139a0f94aa785703528997905cd1febf33015c740ae5626754d61a33ba"} err="failed to get container status \"9f2a67139a0f94aa785703528997905cd1febf33015c740ae5626754d61a33ba\": rpc error: code = NotFound desc = could not find container \"9f2a67139a0f94aa785703528997905cd1febf33015c740ae5626754d61a33ba\": container with ID starting with 9f2a67139a0f94aa785703528997905cd1febf33015c740ae5626754d61a33ba not found: ID does not exist" Sep 08 10:30:50.968640 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.968605 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m"] Sep 08 10:30:50.984256 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:50.984232 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a0d0ba-openshift-default-6db7db7766-pn64m"] Sep 08 10:30:52.678541 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:52.678498 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="308881df-e633-4279-913c-6d19d1aa9274" path="/var/lib/kubelet/pods/308881df-e633-4279-913c-6d19d1aa9274/volumes" Sep 08 10:30:55.723333 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:55.723296 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm"] Sep 08 10:30:55.723903 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:55.723579 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" podUID="57b9899e-6753-4d09-af1c-8e6a9e6c4477" containerName="maas-api" containerID="cri-o://f7f96bfe2cfc4ded38c612e3ca157dbe4f01cc3f1a8639c63f711b3f66ceccc5" gracePeriod=30 Sep 08 10:30:55.756689 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:55.756659 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" podUID="57b9899e-6753-4d09-af1c-8e6a9e6c4477" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.84:8443/health\": dial tcp 10.134.0.84:8443: connect: connection refused" Sep 08 10:30:55.756963 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:55.756936 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:30:55.822286 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:55.822252 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:30:55.907589 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:55.907563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:30:55.967943 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:55.967916 2576 generic.go:356] "Generic (PLEG): container finished" podID="57b9899e-6753-4d09-af1c-8e6a9e6c4477" containerID="f7f96bfe2cfc4ded38c612e3ca157dbe4f01cc3f1a8639c63f711b3f66ceccc5" exitCode=0 Sep 08 10:30:55.968051 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:55.967991 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" event={"ID":"57b9899e-6753-4d09-af1c-8e6a9e6c4477","Type":"ContainerDied","Data":"f7f96bfe2cfc4ded38c612e3ca157dbe4f01cc3f1a8639c63f711b3f66ceccc5"} Sep 08 10:30:55.968051 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:55.968041 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" event={"ID":"57b9899e-6753-4d09-af1c-8e6a9e6c4477","Type":"ContainerDied","Data":"25c11d5e52b6bc43b98a9aa35e62e714d89e2dc4dcb5b914ea4d74ab3074037b"} Sep 08 10:30:55.968136 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:55.968057 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25c11d5e52b6bc43b98a9aa35e62e714d89e2dc4dcb5b914ea4d74ab3074037b" Sep 08 10:30:55.973152 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:55.973126 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:30:55.975813 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:55.975760 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:30:56.045345 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:56.045318 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-827sd\" (UniqueName: \"kubernetes.io/projected/57b9899e-6753-4d09-af1c-8e6a9e6c4477-kube-api-access-827sd\") pod \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " Sep 08 10:30:56.045514 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:56.045382 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57b9899e-6753-4d09-af1c-8e6a9e6c4477-metrics-certs\") pod \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " Sep 08 10:30:56.045514 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:56.045416 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b9899e-6753-4d09-af1c-8e6a9e6c4477-maas-api-tls\") pod \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " Sep 08 10:30:56.045611 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:56.045565 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b9899e-6753-4d09-af1c-8e6a9e6c4477-openshift-service-ca\") pod \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\" (UID: \"57b9899e-6753-4d09-af1c-8e6a9e6c4477\") " Sep 08 10:30:56.045957 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:56.045927 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b9899e-6753-4d09-af1c-8e6a9e6c4477-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57b9899e-6753-4d09-af1c-8e6a9e6c4477" (UID: "57b9899e-6753-4d09-af1c-8e6a9e6c4477"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:30:56.047520 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:56.047499 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57b9899e-6753-4d09-af1c-8e6a9e6c4477-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "57b9899e-6753-4d09-af1c-8e6a9e6c4477" (UID: "57b9899e-6753-4d09-af1c-8e6a9e6c4477"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:30:56.047634 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:56.047518 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57b9899e-6753-4d09-af1c-8e6a9e6c4477-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57b9899e-6753-4d09-af1c-8e6a9e6c4477" (UID: "57b9899e-6753-4d09-af1c-8e6a9e6c4477"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:30:56.047634 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:56.047565 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b9899e-6753-4d09-af1c-8e6a9e6c4477-kube-api-access-827sd" (OuterVolumeSpecName: "kube-api-access-827sd") pod "57b9899e-6753-4d09-af1c-8e6a9e6c4477" (UID: "57b9899e-6753-4d09-af1c-8e6a9e6c4477"). InnerVolumeSpecName "kube-api-access-827sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:30:56.147162 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:56.147128 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-827sd\" (UniqueName: \"kubernetes.io/projected/57b9899e-6753-4d09-af1c-8e6a9e6c4477-kube-api-access-827sd\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:30:56.147162 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:56.147158 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57b9899e-6753-4d09-af1c-8e6a9e6c4477-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:30:56.147162 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:56.147167 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b9899e-6753-4d09-af1c-8e6a9e6c4477-maas-api-tls\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:30:56.147394 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:56.147175 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b9899e-6753-4d09-af1c-8e6a9e6c4477-openshift-service-ca\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:30:56.390339 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:56.390286 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:30:56.974204 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:56.974119 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm" Sep 08 10:30:56.991924 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:56.991891 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm"] Sep 08 10:30:57.001028 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:57.000994 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86902f-8455dbc598-2bgbm"] Sep 08 10:30:58.678313 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:30:58.678277 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57b9899e-6753-4d09-af1c-8e6a9e6c4477" path="/var/lib/kubelet/pods/57b9899e-6753-4d09-af1c-8e6a9e6c4477/volumes" Sep 08 10:31:00.044515 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:00.044481 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:31:02.363915 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:02.363882 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:02.464042 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:02.464008 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:02.537233 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:02.537200 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d"] Sep 08 10:31:02.537461 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:02.537437 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" podUID="9dfd449a-27df-478f-ae15-b3e35398f134" containerName="istio-proxy" containerID="cri-o://0c58360ff58ebb563b1979c0701fb883fe6fba79f1ba84bfcb369f64224b08e0" gracePeriod=30 Sep 08 10:31:02.566554 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:02.566524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:02.819033 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:02.818999 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 10:31:02.819208 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:02.819058 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" podUID="9dfd449a-27df-478f-ae15-b3e35398f134" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 10:31:03.146709 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:03.146675 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:07.795345 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.795321 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:31:07.865087 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865002 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dfd449a-27df-478f-ae15-b3e35398f134-istiod-ca-cert\") pod \"9dfd449a-27df-478f-ae15-b3e35398f134\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " Sep 08 10:31:07.865087 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865043 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-istio-envoy\") pod \"9dfd449a-27df-478f-ae15-b3e35398f134\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " Sep 08 10:31:07.865087 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865065 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfd449a-27df-478f-ae15-b3e35398f134-istio-token\") pod \"9dfd449a-27df-478f-ae15-b3e35398f134\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " Sep 08 10:31:07.865354 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865092 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-credential-socket\") pod \"9dfd449a-27df-478f-ae15-b3e35398f134\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " Sep 08 10:31:07.865354 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865131 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-istio-data\") pod \"9dfd449a-27df-478f-ae15-b3e35398f134\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " Sep 08 10:31:07.865354 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865156 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-workload-certs\") pod \"9dfd449a-27df-478f-ae15-b3e35398f134\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " Sep 08 10:31:07.865354 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865234 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-workload-socket\") pod \"9dfd449a-27df-478f-ae15-b3e35398f134\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " Sep 08 10:31:07.865354 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865272 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dfd449a-27df-478f-ae15-b3e35398f134-istio-podinfo\") pod \"9dfd449a-27df-478f-ae15-b3e35398f134\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " Sep 08 10:31:07.865354 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865351 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s555n\" (UniqueName: \"kubernetes.io/projected/9dfd449a-27df-478f-ae15-b3e35398f134-kube-api-access-s555n\") pod \"9dfd449a-27df-478f-ae15-b3e35398f134\" (UID: \"9dfd449a-27df-478f-ae15-b3e35398f134\") " Sep 08 10:31:07.865703 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865453 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9dfd449a-27df-478f-ae15-b3e35398f134" (UID: "9dfd449a-27df-478f-ae15-b3e35398f134"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:31:07.865703 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865490 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9dfd449a-27df-478f-ae15-b3e35398f134" (UID: "9dfd449a-27df-478f-ae15-b3e35398f134"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:31:07.865703 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865503 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dfd449a-27df-478f-ae15-b3e35398f134-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9dfd449a-27df-478f-ae15-b3e35398f134" (UID: "9dfd449a-27df-478f-ae15-b3e35398f134"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:31:07.865703 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865479 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-istio-data" (OuterVolumeSpecName: "istio-data") pod "9dfd449a-27df-478f-ae15-b3e35398f134" (UID: "9dfd449a-27df-478f-ae15-b3e35398f134"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:31:07.865930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865704 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9dfd449a-27df-478f-ae15-b3e35398f134" (UID: "9dfd449a-27df-478f-ae15-b3e35398f134"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:31:07.865930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865836 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-workload-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:07.865930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865858 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dfd449a-27df-478f-ae15-b3e35398f134-istiod-ca-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:07.865930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865873 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-credential-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:07.865930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865887 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-istio-data\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:07.865930 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.865903 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-workload-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:07.867459 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.867428 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dfd449a-27df-478f-ae15-b3e35398f134-kube-api-access-s555n" (OuterVolumeSpecName: "kube-api-access-s555n") pod "9dfd449a-27df-478f-ae15-b3e35398f134" (UID: "9dfd449a-27df-478f-ae15-b3e35398f134"). InnerVolumeSpecName "kube-api-access-s555n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:31:07.867565 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.867472 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dfd449a-27df-478f-ae15-b3e35398f134-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9dfd449a-27df-478f-ae15-b3e35398f134" (UID: "9dfd449a-27df-478f-ae15-b3e35398f134"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 10:31:07.867653 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.867637 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dfd449a-27df-478f-ae15-b3e35398f134-istio-token" (OuterVolumeSpecName: "istio-token") pod "9dfd449a-27df-478f-ae15-b3e35398f134" (UID: "9dfd449a-27df-478f-ae15-b3e35398f134"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:31:07.867721 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.867705 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9dfd449a-27df-478f-ae15-b3e35398f134" (UID: "9dfd449a-27df-478f-ae15-b3e35398f134"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:31:07.967384 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.967343 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s555n\" (UniqueName: \"kubernetes.io/projected/9dfd449a-27df-478f-ae15-b3e35398f134-kube-api-access-s555n\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:07.967384 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.967376 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dfd449a-27df-478f-ae15-b3e35398f134-istio-envoy\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:07.967384 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.967389 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfd449a-27df-478f-ae15-b3e35398f134-istio-token\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:07.967649 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:07.967401 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dfd449a-27df-478f-ae15-b3e35398f134-istio-podinfo\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:08.038064 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:08.038024 2576 generic.go:356] "Generic (PLEG): container finished" podID="9dfd449a-27df-478f-ae15-b3e35398f134" containerID="0c58360ff58ebb563b1979c0701fb883fe6fba79f1ba84bfcb369f64224b08e0" exitCode=0 Sep 08 10:31:08.038254 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:08.038077 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" Sep 08 10:31:08.038254 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:08.038112 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" event={"ID":"9dfd449a-27df-478f-ae15-b3e35398f134","Type":"ContainerDied","Data":"0c58360ff58ebb563b1979c0701fb883fe6fba79f1ba84bfcb369f64224b08e0"} Sep 08 10:31:08.038254 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:08.038210 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d" event={"ID":"9dfd449a-27df-478f-ae15-b3e35398f134","Type":"ContainerDied","Data":"480220e3d5d176dae6cca2a6f2227be7fd51129dee30bb1dbf6616fc477aeb0e"} Sep 08 10:31:08.038254 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:08.038243 2576 scope.go:117] "RemoveContainer" containerID="0c58360ff58ebb563b1979c0701fb883fe6fba79f1ba84bfcb369f64224b08e0" Sep 08 10:31:08.050627 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:08.050596 2576 scope.go:117] "RemoveContainer" containerID="787a52a8c12607f37d9f3e6b767844f3d40e54fc1f7227858463eee9fa02ef49" Sep 08 10:31:08.059149 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:08.059129 2576 scope.go:117] "RemoveContainer" containerID="0c58360ff58ebb563b1979c0701fb883fe6fba79f1ba84bfcb369f64224b08e0" Sep 08 10:31:08.059418 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:31:08.059398 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c58360ff58ebb563b1979c0701fb883fe6fba79f1ba84bfcb369f64224b08e0\": container with ID starting with 0c58360ff58ebb563b1979c0701fb883fe6fba79f1ba84bfcb369f64224b08e0 not found: ID does not exist" containerID="0c58360ff58ebb563b1979c0701fb883fe6fba79f1ba84bfcb369f64224b08e0" Sep 08 10:31:08.059490 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:08.059431 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c58360ff58ebb563b1979c0701fb883fe6fba79f1ba84bfcb369f64224b08e0"} err="failed to get container status \"0c58360ff58ebb563b1979c0701fb883fe6fba79f1ba84bfcb369f64224b08e0\": rpc error: code = NotFound desc = could not find container \"0c58360ff58ebb563b1979c0701fb883fe6fba79f1ba84bfcb369f64224b08e0\": container with ID starting with 0c58360ff58ebb563b1979c0701fb883fe6fba79f1ba84bfcb369f64224b08e0 not found: ID does not exist" Sep 08 10:31:08.059490 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:08.059454 2576 scope.go:117] "RemoveContainer" containerID="787a52a8c12607f37d9f3e6b767844f3d40e54fc1f7227858463eee9fa02ef49" Sep 08 10:31:08.059770 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:31:08.059750 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"787a52a8c12607f37d9f3e6b767844f3d40e54fc1f7227858463eee9fa02ef49\": container with ID starting with 787a52a8c12607f37d9f3e6b767844f3d40e54fc1f7227858463eee9fa02ef49 not found: ID does not exist" containerID="787a52a8c12607f37d9f3e6b767844f3d40e54fc1f7227858463eee9fa02ef49" Sep 08 10:31:08.059816 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:08.059777 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787a52a8c12607f37d9f3e6b767844f3d40e54fc1f7227858463eee9fa02ef49"} err="failed to get container status \"787a52a8c12607f37d9f3e6b767844f3d40e54fc1f7227858463eee9fa02ef49\": rpc error: code = NotFound desc = could not find container \"787a52a8c12607f37d9f3e6b767844f3d40e54fc1f7227858463eee9fa02ef49\": container with ID starting with 787a52a8c12607f37d9f3e6b767844f3d40e54fc1f7227858463eee9fa02ef49 not found: ID does not exist" Sep 08 10:31:08.066196 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:08.066175 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d"] Sep 08 10:31:08.069978 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:08.069954 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-86902f-openshift-default-fc787d68b-cwb9d"] Sep 08 10:31:08.678368 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:08.678335 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dfd449a-27df-478f-ae15-b3e35398f134" path="/var/lib/kubelet/pods/9dfd449a-27df-478f-ae15-b3e35398f134/volumes" Sep 08 10:31:13.397226 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.397184 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz"] Sep 08 10:31:13.398491 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.398451 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" podUID="8c6dc771-340e-4ee9-ad2f-e90c1aa12245" containerName="maas-api" containerID="cri-o://18f5ca181b0fdf9412427ee8dec9224e0087f2ba2ddc9450331f10dbef159b0b" gracePeriod=30 Sep 08 10:31:13.438784 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.438756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:13.508752 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.508711 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" podUID="8c6dc771-340e-4ee9-ad2f-e90c1aa12245" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.67:8443/health\": dial tcp 10.134.0.67:8443: connect: connection refused" Sep 08 10:31:13.539569 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.539511 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:13.634950 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.634908 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:13.655194 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.655142 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:31:13.703510 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.703478 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:13.725776 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.725745 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-openshift-service-ca\") pod \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " Sep 08 10:31:13.725924 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.725791 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-metrics-certs\") pod \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " Sep 08 10:31:13.725924 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.725894 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwbfv\" (UniqueName: \"kubernetes.io/projected/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-kube-api-access-cwbfv\") pod \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " Sep 08 10:31:13.726058 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.725955 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-maas-api-tls\") pod \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\" (UID: \"8c6dc771-340e-4ee9-ad2f-e90c1aa12245\") " Sep 08 10:31:13.726165 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.726137 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c6dc771-340e-4ee9-ad2f-e90c1aa12245" (UID: "8c6dc771-340e-4ee9-ad2f-e90c1aa12245"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:31:13.726305 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.726275 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-openshift-service-ca\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:13.728205 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.728170 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-kube-api-access-cwbfv" (OuterVolumeSpecName: "kube-api-access-cwbfv") pod "8c6dc771-340e-4ee9-ad2f-e90c1aa12245" (UID: "8c6dc771-340e-4ee9-ad2f-e90c1aa12245"). InnerVolumeSpecName "kube-api-access-cwbfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:31:13.728205 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.728190 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c6dc771-340e-4ee9-ad2f-e90c1aa12245" (UID: "8c6dc771-340e-4ee9-ad2f-e90c1aa12245"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:31:13.728351 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.728207 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8c6dc771-340e-4ee9-ad2f-e90c1aa12245" (UID: "8c6dc771-340e-4ee9-ad2f-e90c1aa12245"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:31:13.827293 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.827245 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-maas-api-tls\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:13.827293 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.827289 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:13.827505 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:13.827305 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwbfv\" (UniqueName: \"kubernetes.io/projected/8c6dc771-340e-4ee9-ad2f-e90c1aa12245-kube-api-access-cwbfv\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:14.066507 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:14.066420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:14.076421 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:14.076389 2576 generic.go:356] "Generic (PLEG): container finished" podID="8c6dc771-340e-4ee9-ad2f-e90c1aa12245" containerID="18f5ca181b0fdf9412427ee8dec9224e0087f2ba2ddc9450331f10dbef159b0b" exitCode=0 Sep 08 10:31:14.076585 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:14.076440 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" Sep 08 10:31:14.076585 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:14.076454 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" event={"ID":"8c6dc771-340e-4ee9-ad2f-e90c1aa12245","Type":"ContainerDied","Data":"18f5ca181b0fdf9412427ee8dec9224e0087f2ba2ddc9450331f10dbef159b0b"} Sep 08 10:31:14.076585 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:14.076505 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz" event={"ID":"8c6dc771-340e-4ee9-ad2f-e90c1aa12245","Type":"ContainerDied","Data":"2508becd552d191539a2a64e73813394ab21f1b1f5887aa347da72fe33f7a09e"} Sep 08 10:31:14.076585 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:14.076535 2576 scope.go:117] "RemoveContainer" containerID="18f5ca181b0fdf9412427ee8dec9224e0087f2ba2ddc9450331f10dbef159b0b" Sep 08 10:31:14.086274 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:14.086258 2576 scope.go:117] "RemoveContainer" containerID="18f5ca181b0fdf9412427ee8dec9224e0087f2ba2ddc9450331f10dbef159b0b" Sep 08 10:31:14.086553 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:31:14.086535 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f5ca181b0fdf9412427ee8dec9224e0087f2ba2ddc9450331f10dbef159b0b\": container with ID starting with 18f5ca181b0fdf9412427ee8dec9224e0087f2ba2ddc9450331f10dbef159b0b not found: ID does not exist" containerID="18f5ca181b0fdf9412427ee8dec9224e0087f2ba2ddc9450331f10dbef159b0b" Sep 08 10:31:14.086639 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:14.086559 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f5ca181b0fdf9412427ee8dec9224e0087f2ba2ddc9450331f10dbef159b0b"} err="failed to get container status \"18f5ca181b0fdf9412427ee8dec9224e0087f2ba2ddc9450331f10dbef159b0b\": rpc error: code = NotFound desc = could not find container \"18f5ca181b0fdf9412427ee8dec9224e0087f2ba2ddc9450331f10dbef159b0b\": container with ID starting with 18f5ca181b0fdf9412427ee8dec9224e0087f2ba2ddc9450331f10dbef159b0b not found: ID does not exist" Sep 08 10:31:14.106595 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:14.106564 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz"] Sep 08 10:31:14.120471 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:14.120445 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-71a7a1-6b556f6789-24pxz"] Sep 08 10:31:14.677522 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:14.677487 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c6dc771-340e-4ee9-ad2f-e90c1aa12245" path="/var/lib/kubelet/pods/8c6dc771-340e-4ee9-ad2f-e90c1aa12245/volumes" Sep 08 10:31:18.250595 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:18.250562 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:31:19.908183 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:19.908149 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:19.985390 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:19.985357 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:20.164974 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:20.164890 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh"] Sep 08 10:31:20.165210 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:20.165185 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" podUID="26271269-74fa-4615-bb71-085a510fe223" containerName="istio-proxy" containerID="cri-o://4f6de6638660118d155810c99ce06b6bd84141d71cc294b537d4bd5c4790f625" gracePeriod=30 Sep 08 10:31:20.187034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:20.187002 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:20.603063 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:20.602983 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:21.860478 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:21.860439 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 10:31:21.860857 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:21.860506 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" podUID="26271269-74fa-4615-bb71-085a510fe223" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 10:31:23.417441 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:23.417363 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:25.424678 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.424653 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:31:25.546531 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.546430 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-workload-certs\") pod \"26271269-74fa-4615-bb71-085a510fe223\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " Sep 08 10:31:25.546531 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.546480 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krksw\" (UniqueName: \"kubernetes.io/projected/26271269-74fa-4615-bb71-085a510fe223-kube-api-access-krksw\") pod \"26271269-74fa-4615-bb71-085a510fe223\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " Sep 08 10:31:25.546531 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.546530 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-workload-socket\") pod \"26271269-74fa-4615-bb71-085a510fe223\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " Sep 08 10:31:25.546879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.546558 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-istio-envoy\") pod \"26271269-74fa-4615-bb71-085a510fe223\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " Sep 08 10:31:25.546879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.546584 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-istio-data\") pod \"26271269-74fa-4615-bb71-085a510fe223\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " Sep 08 10:31:25.546879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.546701 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26271269-74fa-4615-bb71-085a510fe223-istiod-ca-cert\") pod \"26271269-74fa-4615-bb71-085a510fe223\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " Sep 08 10:31:25.546879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.546731 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26271269-74fa-4615-bb71-085a510fe223-istio-podinfo\") pod \"26271269-74fa-4615-bb71-085a510fe223\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " Sep 08 10:31:25.546879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.546768 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "26271269-74fa-4615-bb71-085a510fe223" (UID: "26271269-74fa-4615-bb71-085a510fe223"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:31:25.547157 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.546884 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "26271269-74fa-4615-bb71-085a510fe223" (UID: "26271269-74fa-4615-bb71-085a510fe223"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:31:25.547157 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.546899 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-istio-data" (OuterVolumeSpecName: "istio-data") pod "26271269-74fa-4615-bb71-085a510fe223" (UID: "26271269-74fa-4615-bb71-085a510fe223"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:31:25.547157 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.547082 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26271269-74fa-4615-bb71-085a510fe223-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "26271269-74fa-4615-bb71-085a510fe223" (UID: "26271269-74fa-4615-bb71-085a510fe223"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:31:25.547373 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.547352 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26271269-74fa-4615-bb71-085a510fe223-istio-token\") pod \"26271269-74fa-4615-bb71-085a510fe223\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " Sep 08 10:31:25.547467 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.547403 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-credential-socket\") pod \"26271269-74fa-4615-bb71-085a510fe223\" (UID: \"26271269-74fa-4615-bb71-085a510fe223\") " Sep 08 10:31:25.547749 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.547714 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "26271269-74fa-4615-bb71-085a510fe223" (UID: "26271269-74fa-4615-bb71-085a510fe223"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:31:25.547888 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.547861 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-credential-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:25.547888 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.547883 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-workload-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:25.547980 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.547897 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-workload-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:25.547980 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.547909 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-istio-data\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:25.547980 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.547922 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26271269-74fa-4615-bb71-085a510fe223-istiod-ca-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:25.548944 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.548919 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26271269-74fa-4615-bb71-085a510fe223-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "26271269-74fa-4615-bb71-085a510fe223" (UID: "26271269-74fa-4615-bb71-085a510fe223"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 10:31:25.549024 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.549008 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26271269-74fa-4615-bb71-085a510fe223-kube-api-access-krksw" (OuterVolumeSpecName: "kube-api-access-krksw") pod "26271269-74fa-4615-bb71-085a510fe223" (UID: "26271269-74fa-4615-bb71-085a510fe223"). InnerVolumeSpecName "kube-api-access-krksw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:31:25.549194 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.549169 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "26271269-74fa-4615-bb71-085a510fe223" (UID: "26271269-74fa-4615-bb71-085a510fe223"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:31:25.549421 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.549408 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26271269-74fa-4615-bb71-085a510fe223-istio-token" (OuterVolumeSpecName: "istio-token") pod "26271269-74fa-4615-bb71-085a510fe223" (UID: "26271269-74fa-4615-bb71-085a510fe223"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:31:25.648872 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.648832 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26271269-74fa-4615-bb71-085a510fe223-istio-envoy\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:25.648872 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.648865 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26271269-74fa-4615-bb71-085a510fe223-istio-podinfo\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:25.648872 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.648874 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26271269-74fa-4615-bb71-085a510fe223-istio-token\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:25.649120 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:25.648890 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krksw\" (UniqueName: \"kubernetes.io/projected/26271269-74fa-4615-bb71-085a510fe223-kube-api-access-krksw\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:26.156824 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:26.156786 2576 generic.go:356] "Generic (PLEG): container finished" podID="26271269-74fa-4615-bb71-085a510fe223" containerID="4f6de6638660118d155810c99ce06b6bd84141d71cc294b537d4bd5c4790f625" exitCode=0 Sep 08 10:31:26.157052 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:26.156838 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" event={"ID":"26271269-74fa-4615-bb71-085a510fe223","Type":"ContainerDied","Data":"4f6de6638660118d155810c99ce06b6bd84141d71cc294b537d4bd5c4790f625"} Sep 08 10:31:26.157052 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:26.156856 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" Sep 08 10:31:26.157052 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:26.156912 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh" event={"ID":"26271269-74fa-4615-bb71-085a510fe223","Type":"ContainerDied","Data":"74dca75be1c53753533597c48e70f4a274ab8c29f47ee706593d35576eb03f17"} Sep 08 10:31:26.157052 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:26.156934 2576 scope.go:117] "RemoveContainer" containerID="4f6de6638660118d155810c99ce06b6bd84141d71cc294b537d4bd5c4790f625" Sep 08 10:31:26.166664 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:26.166643 2576 scope.go:117] "RemoveContainer" containerID="9bc2a5b6f0923d23161acd4dc7f085434fedaabbae50d0f609b41291fda55d84" Sep 08 10:31:26.175171 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:26.175147 2576 scope.go:117] "RemoveContainer" containerID="4f6de6638660118d155810c99ce06b6bd84141d71cc294b537d4bd5c4790f625" Sep 08 10:31:26.175459 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:31:26.175440 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f6de6638660118d155810c99ce06b6bd84141d71cc294b537d4bd5c4790f625\": container with ID starting with 4f6de6638660118d155810c99ce06b6bd84141d71cc294b537d4bd5c4790f625 not found: ID does not exist" containerID="4f6de6638660118d155810c99ce06b6bd84141d71cc294b537d4bd5c4790f625" Sep 08 10:31:26.175499 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:26.175469 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6de6638660118d155810c99ce06b6bd84141d71cc294b537d4bd5c4790f625"} err="failed to get container status \"4f6de6638660118d155810c99ce06b6bd84141d71cc294b537d4bd5c4790f625\": rpc error: code = NotFound desc = could not find container \"4f6de6638660118d155810c99ce06b6bd84141d71cc294b537d4bd5c4790f625\": container with ID starting with 4f6de6638660118d155810c99ce06b6bd84141d71cc294b537d4bd5c4790f625 not found: ID does not exist" Sep 08 10:31:26.175499 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:26.175486 2576 scope.go:117] "RemoveContainer" containerID="9bc2a5b6f0923d23161acd4dc7f085434fedaabbae50d0f609b41291fda55d84" Sep 08 10:31:26.175784 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:31:26.175766 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc2a5b6f0923d23161acd4dc7f085434fedaabbae50d0f609b41291fda55d84\": container with ID starting with 9bc2a5b6f0923d23161acd4dc7f085434fedaabbae50d0f609b41291fda55d84 not found: ID does not exist" containerID="9bc2a5b6f0923d23161acd4dc7f085434fedaabbae50d0f609b41291fda55d84" Sep 08 10:31:26.175831 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:26.175793 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc2a5b6f0923d23161acd4dc7f085434fedaabbae50d0f609b41291fda55d84"} err="failed to get container status \"9bc2a5b6f0923d23161acd4dc7f085434fedaabbae50d0f609b41291fda55d84\": rpc error: code = NotFound desc = could not find container \"9bc2a5b6f0923d23161acd4dc7f085434fedaabbae50d0f609b41291fda55d84\": container with ID starting with 9bc2a5b6f0923d23161acd4dc7f085434fedaabbae50d0f609b41291fda55d84 not found: ID does not exist" Sep 08 10:31:26.181879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:26.181857 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh"] Sep 08 10:31:26.185334 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:26.185311 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-71a7a1-openshift-default-55c6d8c444-smjhh"] Sep 08 10:31:26.678702 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:26.678672 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26271269-74fa-4615-bb71-085a510fe223" path="/var/lib/kubelet/pods/26271269-74fa-4615-bb71-085a510fe223/volumes" Sep 08 10:31:30.649338 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:30.649294 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:31.243815 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:31.243772 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:31.328572 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:31.328525 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:31.330301 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:31.330279 2576 scope.go:117] "RemoveContainer" containerID="c5ef6d8c43c688f9d672364a835c575b97b5d0c62e7054f7368ba07425336112" Sep 08 10:31:31.478487 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:31.478456 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:31.614699 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:31.614595 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:31.919894 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:31.919858 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:34.462598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:34.462565 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:31:39.083116 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:39.083087 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:39.182212 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:39.182177 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:39.307527 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:39.307497 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j"] Sep 08 10:31:39.307748 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:39.307723 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" podUID="871d1b48-9d6d-490e-aa12-14329ffef7a9" containerName="istio-proxy" containerID="cri-o://ad506f8e7a1c7dbd3742ad7410ecb43038a59e5cbd79f9a2496c48b5ab0999d3" gracePeriod=30 Sep 08 10:31:39.342179 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:39.342109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:31:39.388725 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:39.388698 2576 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 10:31:39.388956 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:39.388922 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" podUID="871d1b48-9d6d-490e-aa12-14329ffef7a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 10:31:41.950506 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:41.950472 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-869d99c85c-sfn7r"] Sep 08 10:31:41.950887 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:41.950689 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-869d99c85c-sfn7r" podUID="69e54261-e36c-45ce-99cb-d2af888add0b" containerName="manager" containerID="cri-o://9a79419ec943e746adf5195a2ba892a872fa31f090eda54aa8622fad056957d3" gracePeriod=30 Sep 08 10:31:42.208651 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.208573 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-869d99c85c-sfn7r" Sep 08 10:31:42.251139 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.251100 2576 generic.go:356] "Generic (PLEG): container finished" podID="69e54261-e36c-45ce-99cb-d2af888add0b" containerID="9a79419ec943e746adf5195a2ba892a872fa31f090eda54aa8622fad056957d3" exitCode=0 Sep 08 10:31:42.251292 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.251159 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-869d99c85c-sfn7r" Sep 08 10:31:42.251292 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.251174 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-869d99c85c-sfn7r" event={"ID":"69e54261-e36c-45ce-99cb-d2af888add0b","Type":"ContainerDied","Data":"9a79419ec943e746adf5195a2ba892a872fa31f090eda54aa8622fad056957d3"} Sep 08 10:31:42.251292 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.251247 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-869d99c85c-sfn7r" event={"ID":"69e54261-e36c-45ce-99cb-d2af888add0b","Type":"ContainerDied","Data":"7c5db6dbf5b73de29b415dfd135851bb4acca264cd0289f90032aacad53e29b2"} Sep 08 10:31:42.251292 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.251276 2576 scope.go:117] "RemoveContainer" containerID="9a79419ec943e746adf5195a2ba892a872fa31f090eda54aa8622fad056957d3" Sep 08 10:31:42.260967 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.260949 2576 scope.go:117] "RemoveContainer" containerID="9a79419ec943e746adf5195a2ba892a872fa31f090eda54aa8622fad056957d3" Sep 08 10:31:42.261253 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:31:42.261235 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a79419ec943e746adf5195a2ba892a872fa31f090eda54aa8622fad056957d3\": container with ID starting with 9a79419ec943e746adf5195a2ba892a872fa31f090eda54aa8622fad056957d3 not found: ID does not exist" containerID="9a79419ec943e746adf5195a2ba892a872fa31f090eda54aa8622fad056957d3" Sep 08 10:31:42.261335 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.261261 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a79419ec943e746adf5195a2ba892a872fa31f090eda54aa8622fad056957d3"} err="failed to get container status \"9a79419ec943e746adf5195a2ba892a872fa31f090eda54aa8622fad056957d3\": rpc error: code = NotFound desc = could not find container \"9a79419ec943e746adf5195a2ba892a872fa31f090eda54aa8622fad056957d3\": container with ID starting with 9a79419ec943e746adf5195a2ba892a872fa31f090eda54aa8622fad056957d3 not found: ID does not exist" Sep 08 10:31:42.287854 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.287831 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69e54261-e36c-45ce-99cb-d2af888add0b-metrics-certs\") pod \"69e54261-e36c-45ce-99cb-d2af888add0b\" (UID: \"69e54261-e36c-45ce-99cb-d2af888add0b\") " Sep 08 10:31:42.287977 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.287880 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69e54261-e36c-45ce-99cb-d2af888add0b-webhook-cert\") pod \"69e54261-e36c-45ce-99cb-d2af888add0b\" (UID: \"69e54261-e36c-45ce-99cb-d2af888add0b\") " Sep 08 10:31:42.287977 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.287931 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnt68\" (UniqueName: \"kubernetes.io/projected/69e54261-e36c-45ce-99cb-d2af888add0b-kube-api-access-xnt68\") pod \"69e54261-e36c-45ce-99cb-d2af888add0b\" (UID: \"69e54261-e36c-45ce-99cb-d2af888add0b\") " Sep 08 10:31:42.289970 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.289941 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e54261-e36c-45ce-99cb-d2af888add0b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "69e54261-e36c-45ce-99cb-d2af888add0b" (UID: "69e54261-e36c-45ce-99cb-d2af888add0b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:31:42.290192 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.290165 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e54261-e36c-45ce-99cb-d2af888add0b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "69e54261-e36c-45ce-99cb-d2af888add0b" (UID: "69e54261-e36c-45ce-99cb-d2af888add0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:31:42.290192 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.290167 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e54261-e36c-45ce-99cb-d2af888add0b-kube-api-access-xnt68" (OuterVolumeSpecName: "kube-api-access-xnt68") pod "69e54261-e36c-45ce-99cb-d2af888add0b" (UID: "69e54261-e36c-45ce-99cb-d2af888add0b"). InnerVolumeSpecName "kube-api-access-xnt68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:31:42.389258 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.389230 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69e54261-e36c-45ce-99cb-d2af888add0b-webhook-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:42.389258 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.389253 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnt68\" (UniqueName: \"kubernetes.io/projected/69e54261-e36c-45ce-99cb-d2af888add0b-kube-api-access-xnt68\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:42.389460 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.389266 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69e54261-e36c-45ce-99cb-d2af888add0b-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:42.579922 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.579886 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-869d99c85c-sfn7r"] Sep 08 10:31:42.583863 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.583840 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-869d99c85c-sfn7r"] Sep 08 10:31:42.677193 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:42.677162 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69e54261-e36c-45ce-99cb-d2af888add0b" path="/var/lib/kubelet/pods/69e54261-e36c-45ce-99cb-d2af888add0b/volumes" Sep 08 10:31:43.454186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:43.454147 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:31:44.560491 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.560468 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:31:44.609522 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.609452 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-envoy\") pod \"871d1b48-9d6d-490e-aa12-14329ffef7a9\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " Sep 08 10:31:44.609522 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.609483 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-data\") pod \"871d1b48-9d6d-490e-aa12-14329ffef7a9\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " Sep 08 10:31:44.609522 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.609507 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-credential-socket\") pod \"871d1b48-9d6d-490e-aa12-14329ffef7a9\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " Sep 08 10:31:44.609818 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.609530 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdnfw\" (UniqueName: \"kubernetes.io/projected/871d1b48-9d6d-490e-aa12-14329ffef7a9-kube-api-access-fdnfw\") pod \"871d1b48-9d6d-490e-aa12-14329ffef7a9\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " Sep 08 10:31:44.609818 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.609557 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-token\") pod \"871d1b48-9d6d-490e-aa12-14329ffef7a9\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " Sep 08 10:31:44.609818 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.609631 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-workload-socket\") pod \"871d1b48-9d6d-490e-aa12-14329ffef7a9\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " Sep 08 10:31:44.609818 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.609672 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-workload-certs\") pod \"871d1b48-9d6d-490e-aa12-14329ffef7a9\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " Sep 08 10:31:44.609818 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.609721 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-podinfo\") pod \"871d1b48-9d6d-490e-aa12-14329ffef7a9\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " Sep 08 10:31:44.609818 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.609743 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871d1b48-9d6d-490e-aa12-14329ffef7a9-istiod-ca-cert\") pod \"871d1b48-9d6d-490e-aa12-14329ffef7a9\" (UID: \"871d1b48-9d6d-490e-aa12-14329ffef7a9\") " Sep 08 10:31:44.610143 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.609886 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "871d1b48-9d6d-490e-aa12-14329ffef7a9" (UID: "871d1b48-9d6d-490e-aa12-14329ffef7a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:31:44.610143 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.610013 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "871d1b48-9d6d-490e-aa12-14329ffef7a9" (UID: "871d1b48-9d6d-490e-aa12-14329ffef7a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:31:44.610143 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.610064 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-credential-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:44.610307 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.610279 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-data" (OuterVolumeSpecName: "istio-data") pod "871d1b48-9d6d-490e-aa12-14329ffef7a9" (UID: "871d1b48-9d6d-490e-aa12-14329ffef7a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:31:44.610377 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.610301 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "871d1b48-9d6d-490e-aa12-14329ffef7a9" (UID: "871d1b48-9d6d-490e-aa12-14329ffef7a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:31:44.610377 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.610329 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871d1b48-9d6d-490e-aa12-14329ffef7a9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "871d1b48-9d6d-490e-aa12-14329ffef7a9" (UID: "871d1b48-9d6d-490e-aa12-14329ffef7a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:31:44.611944 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.611910 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871d1b48-9d6d-490e-aa12-14329ffef7a9-kube-api-access-fdnfw" (OuterVolumeSpecName: "kube-api-access-fdnfw") pod "871d1b48-9d6d-490e-aa12-14329ffef7a9" (UID: "871d1b48-9d6d-490e-aa12-14329ffef7a9"). InnerVolumeSpecName "kube-api-access-fdnfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:31:44.612042 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.611941 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "871d1b48-9d6d-490e-aa12-14329ffef7a9" (UID: "871d1b48-9d6d-490e-aa12-14329ffef7a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 10:31:44.612305 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.612286 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "871d1b48-9d6d-490e-aa12-14329ffef7a9" (UID: "871d1b48-9d6d-490e-aa12-14329ffef7a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:31:44.612487 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.612466 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-token" (OuterVolumeSpecName: "istio-token") pod "871d1b48-9d6d-490e-aa12-14329ffef7a9" (UID: "871d1b48-9d6d-490e-aa12-14329ffef7a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:31:44.711207 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.711180 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-token\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:44.711346 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.711212 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-workload-socket\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:44.711346 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.711228 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-workload-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:44.711346 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.711243 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-podinfo\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:44.711346 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.711256 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871d1b48-9d6d-490e-aa12-14329ffef7a9-istiod-ca-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:44.711346 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.711270 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-envoy\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:44.711346 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.711283 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871d1b48-9d6d-490e-aa12-14329ffef7a9-istio-data\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:44.711346 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:44.711296 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdnfw\" (UniqueName: \"kubernetes.io/projected/871d1b48-9d6d-490e-aa12-14329ffef7a9-kube-api-access-fdnfw\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:31:45.270697 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:45.270663 2576 generic.go:356] "Generic (PLEG): container finished" podID="871d1b48-9d6d-490e-aa12-14329ffef7a9" containerID="ad506f8e7a1c7dbd3742ad7410ecb43038a59e5cbd79f9a2496c48b5ab0999d3" exitCode=0 Sep 08 10:31:45.270878 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:45.270712 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" Sep 08 10:31:45.270878 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:45.270758 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" event={"ID":"871d1b48-9d6d-490e-aa12-14329ffef7a9","Type":"ContainerDied","Data":"ad506f8e7a1c7dbd3742ad7410ecb43038a59e5cbd79f9a2496c48b5ab0999d3"} Sep 08 10:31:45.270878 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:45.270816 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j" event={"ID":"871d1b48-9d6d-490e-aa12-14329ffef7a9","Type":"ContainerDied","Data":"8774dfa92947dd6202c365cc210b240810b4100eca3f9b66cc58e126a9c7a23c"} Sep 08 10:31:45.270878 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:45.270841 2576 scope.go:117] "RemoveContainer" containerID="ad506f8e7a1c7dbd3742ad7410ecb43038a59e5cbd79f9a2496c48b5ab0999d3" Sep 08 10:31:45.279750 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:45.279736 2576 scope.go:117] "RemoveContainer" containerID="afe771bcd09a4750b054bbb8f306cd8678e1ab9d356b6542dc72253aab0e668d" Sep 08 10:31:45.287978 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:45.287963 2576 scope.go:117] "RemoveContainer" containerID="ad506f8e7a1c7dbd3742ad7410ecb43038a59e5cbd79f9a2496c48b5ab0999d3" Sep 08 10:31:45.288222 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:31:45.288203 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad506f8e7a1c7dbd3742ad7410ecb43038a59e5cbd79f9a2496c48b5ab0999d3\": container with ID starting with ad506f8e7a1c7dbd3742ad7410ecb43038a59e5cbd79f9a2496c48b5ab0999d3 not found: ID does not exist" containerID="ad506f8e7a1c7dbd3742ad7410ecb43038a59e5cbd79f9a2496c48b5ab0999d3" Sep 08 10:31:45.288289 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:45.288228 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad506f8e7a1c7dbd3742ad7410ecb43038a59e5cbd79f9a2496c48b5ab0999d3"} err="failed to get container status \"ad506f8e7a1c7dbd3742ad7410ecb43038a59e5cbd79f9a2496c48b5ab0999d3\": rpc error: code = NotFound desc = could not find container \"ad506f8e7a1c7dbd3742ad7410ecb43038a59e5cbd79f9a2496c48b5ab0999d3\": container with ID starting with ad506f8e7a1c7dbd3742ad7410ecb43038a59e5cbd79f9a2496c48b5ab0999d3 not found: ID does not exist" Sep 08 10:31:45.288289 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:45.288243 2576 scope.go:117] "RemoveContainer" containerID="afe771bcd09a4750b054bbb8f306cd8678e1ab9d356b6542dc72253aab0e668d" Sep 08 10:31:45.288470 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:31:45.288447 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe771bcd09a4750b054bbb8f306cd8678e1ab9d356b6542dc72253aab0e668d\": container with ID starting with afe771bcd09a4750b054bbb8f306cd8678e1ab9d356b6542dc72253aab0e668d not found: ID does not exist" containerID="afe771bcd09a4750b054bbb8f306cd8678e1ab9d356b6542dc72253aab0e668d" Sep 08 10:31:45.288528 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:45.288475 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe771bcd09a4750b054bbb8f306cd8678e1ab9d356b6542dc72253aab0e668d"} err="failed to get container status \"afe771bcd09a4750b054bbb8f306cd8678e1ab9d356b6542dc72253aab0e668d\": rpc error: code = NotFound desc = could not find container \"afe771bcd09a4750b054bbb8f306cd8678e1ab9d356b6542dc72253aab0e668d\": container with ID starting with afe771bcd09a4750b054bbb8f306cd8678e1ab9d356b6542dc72253aab0e668d not found: ID does not exist" Sep 08 10:31:45.289530 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:45.289507 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j"] Sep 08 10:31:45.295011 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:45.294991 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d6e53c-openshift-default-7fc5b8cf58-prz2j"] Sep 08 10:31:46.678043 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:31:46.678010 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="871d1b48-9d6d-490e-aa12-14329ffef7a9" path="/var/lib/kubelet/pods/871d1b48-9d6d-490e-aa12-14329ffef7a9/volumes" Sep 08 10:32:25.521926 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:25.521888 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:32:30.413067 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:30.413036 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:32:43.473699 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.473660 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-869d99c85c-x978q"] Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474051 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="308881df-e633-4279-913c-6d19d1aa9274" containerName="istio-proxy" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474062 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="308881df-e633-4279-913c-6d19d1aa9274" containerName="istio-proxy" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474069 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23675580-e81a-4cc3-9539-9e109879d0c9" containerName="maas-api" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474075 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23675580-e81a-4cc3-9539-9e109879d0c9" containerName="maas-api" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474086 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="871d1b48-9d6d-490e-aa12-14329ffef7a9" containerName="istio-proxy" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474092 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="871d1b48-9d6d-490e-aa12-14329ffef7a9" containerName="istio-proxy" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474099 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57b9899e-6753-4d09-af1c-8e6a9e6c4477" containerName="maas-api" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474104 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="57b9899e-6753-4d09-af1c-8e6a9e6c4477" containerName="maas-api" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474111 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c6dc771-340e-4ee9-ad2f-e90c1aa12245" containerName="maas-api" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474116 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c6dc771-340e-4ee9-ad2f-e90c1aa12245" containerName="maas-api" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474128 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dfd449a-27df-478f-ae15-b3e35398f134" containerName="istio-proxy" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474133 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dfd449a-27df-478f-ae15-b3e35398f134" containerName="istio-proxy" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474139 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="871d1b48-9d6d-490e-aa12-14329ffef7a9" containerName="istio-proxy" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474144 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="871d1b48-9d6d-490e-aa12-14329ffef7a9" containerName="istio-proxy" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474155 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e54261-e36c-45ce-99cb-d2af888add0b" containerName="manager" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474160 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e54261-e36c-45ce-99cb-d2af888add0b" containerName="manager" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474168 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="308881df-e633-4279-913c-6d19d1aa9274" containerName="istio-proxy" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474173 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="308881df-e633-4279-913c-6d19d1aa9274" containerName="istio-proxy" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474179 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dfd449a-27df-478f-ae15-b3e35398f134" containerName="istio-proxy" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474185 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dfd449a-27df-478f-ae15-b3e35398f134" containerName="istio-proxy" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474192 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26271269-74fa-4615-bb71-085a510fe223" containerName="istio-proxy" Sep 08 10:32:43.474186 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474197 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="26271269-74fa-4615-bb71-085a510fe223" containerName="istio-proxy" Sep 08 10:32:43.474879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474207 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26271269-74fa-4615-bb71-085a510fe223" containerName="istio-proxy" Sep 08 10:32:43.474879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474212 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="26271269-74fa-4615-bb71-085a510fe223" containerName="istio-proxy" Sep 08 10:32:43.474879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474263 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="26271269-74fa-4615-bb71-085a510fe223" containerName="istio-proxy" Sep 08 10:32:43.474879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474270 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dfd449a-27df-478f-ae15-b3e35398f134" containerName="istio-proxy" Sep 08 10:32:43.474879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474276 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="69e54261-e36c-45ce-99cb-d2af888add0b" containerName="manager" Sep 08 10:32:43.474879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474283 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="871d1b48-9d6d-490e-aa12-14329ffef7a9" containerName="istio-proxy" Sep 08 10:32:43.474879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474289 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="23675580-e81a-4cc3-9539-9e109879d0c9" containerName="maas-api" Sep 08 10:32:43.474879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474294 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c6dc771-340e-4ee9-ad2f-e90c1aa12245" containerName="maas-api" Sep 08 10:32:43.474879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474300 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="57b9899e-6753-4d09-af1c-8e6a9e6c4477" containerName="maas-api" Sep 08 10:32:43.474879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474305 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="308881df-e633-4279-913c-6d19d1aa9274" containerName="istio-proxy" Sep 08 10:32:43.474879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474310 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="871d1b48-9d6d-490e-aa12-14329ffef7a9" containerName="istio-proxy" Sep 08 10:32:43.474879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.474315 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dfd449a-27df-478f-ae15-b3e35398f134" containerName="istio-proxy" Sep 08 10:32:43.477299 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.477284 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-869d99c85c-x978q" Sep 08 10:32:43.481034 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.481013 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nf7fl\"" Sep 08 10:32:43.481841 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.481824 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 10:32:43.481940 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.481928 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 10:32:43.482005 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.481976 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 10:32:43.504050 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.504023 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-869d99c85c-x978q"] Sep 08 10:32:43.628142 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.628104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d49494c1-dc53-44ad-a095-2bca054df5e1-metrics-certs\") pod \"maas-controller-869d99c85c-x978q\" (UID: \"d49494c1-dc53-44ad-a095-2bca054df5e1\") " pod="opendatahub/maas-controller-869d99c85c-x978q" Sep 08 10:32:43.628319 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.628174 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d49494c1-dc53-44ad-a095-2bca054df5e1-webhook-cert\") pod \"maas-controller-869d99c85c-x978q\" (UID: \"d49494c1-dc53-44ad-a095-2bca054df5e1\") " pod="opendatahub/maas-controller-869d99c85c-x978q" Sep 08 10:32:43.628319 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.628229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkvzf\" (UniqueName: \"kubernetes.io/projected/d49494c1-dc53-44ad-a095-2bca054df5e1-kube-api-access-jkvzf\") pod \"maas-controller-869d99c85c-x978q\" (UID: \"d49494c1-dc53-44ad-a095-2bca054df5e1\") " pod="opendatahub/maas-controller-869d99c85c-x978q" Sep 08 10:32:43.729205 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.729103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d49494c1-dc53-44ad-a095-2bca054df5e1-metrics-certs\") pod \"maas-controller-869d99c85c-x978q\" (UID: \"d49494c1-dc53-44ad-a095-2bca054df5e1\") " pod="opendatahub/maas-controller-869d99c85c-x978q" Sep 08 10:32:43.729205 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.729205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d49494c1-dc53-44ad-a095-2bca054df5e1-webhook-cert\") pod \"maas-controller-869d99c85c-x978q\" (UID: \"d49494c1-dc53-44ad-a095-2bca054df5e1\") " pod="opendatahub/maas-controller-869d99c85c-x978q" Sep 08 10:32:43.729454 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.729376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkvzf\" (UniqueName: \"kubernetes.io/projected/d49494c1-dc53-44ad-a095-2bca054df5e1-kube-api-access-jkvzf\") pod \"maas-controller-869d99c85c-x978q\" (UID: \"d49494c1-dc53-44ad-a095-2bca054df5e1\") " pod="opendatahub/maas-controller-869d99c85c-x978q" Sep 08 10:32:43.731592 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.731562 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d49494c1-dc53-44ad-a095-2bca054df5e1-metrics-certs\") pod \"maas-controller-869d99c85c-x978q\" (UID: \"d49494c1-dc53-44ad-a095-2bca054df5e1\") " pod="opendatahub/maas-controller-869d99c85c-x978q" Sep 08 10:32:43.731726 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.731589 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d49494c1-dc53-44ad-a095-2bca054df5e1-webhook-cert\") pod \"maas-controller-869d99c85c-x978q\" (UID: \"d49494c1-dc53-44ad-a095-2bca054df5e1\") " pod="opendatahub/maas-controller-869d99c85c-x978q" Sep 08 10:32:43.739510 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.739486 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkvzf\" (UniqueName: \"kubernetes.io/projected/d49494c1-dc53-44ad-a095-2bca054df5e1-kube-api-access-jkvzf\") pod \"maas-controller-869d99c85c-x978q\" (UID: \"d49494c1-dc53-44ad-a095-2bca054df5e1\") " pod="opendatahub/maas-controller-869d99c85c-x978q" Sep 08 10:32:43.787003 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:43.786974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-869d99c85c-x978q" Sep 08 10:32:44.061009 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:44.060984 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-869d99c85c-x978q"] Sep 08 10:32:44.062612 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:32:44.062583 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49494c1_dc53_44ad_a095_2bca054df5e1.slice/crio-4d00d261e03654506e2f18ee1791fb645deb2ef6a5de388d45a18813e733a0e5 WatchSource:0}: Error finding container 4d00d261e03654506e2f18ee1791fb645deb2ef6a5de388d45a18813e733a0e5: Status 404 returned error can't find the container with id 4d00d261e03654506e2f18ee1791fb645deb2ef6a5de388d45a18813e733a0e5 Sep 08 10:32:44.063975 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:44.063959 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 10:32:44.600675 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:44.600644 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-869d99c85c-x978q" event={"ID":"d49494c1-dc53-44ad-a095-2bca054df5e1","Type":"ContainerStarted","Data":"4d00d261e03654506e2f18ee1791fb645deb2ef6a5de388d45a18813e733a0e5"} Sep 08 10:32:45.607886 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:45.607849 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-869d99c85c-x978q" event={"ID":"d49494c1-dc53-44ad-a095-2bca054df5e1","Type":"ContainerStarted","Data":"49c28c88956fcc7007340dd86f6a2346042952970f9d188e091d60850067c230"} Sep 08 10:32:45.608271 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:45.607983 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-869d99c85c-x978q" Sep 08 10:32:45.626179 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:45.626129 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-869d99c85c-x978q" podStartSLOduration=2.130358895 podStartE2EDuration="2.626113892s" totalImagesPullingTime="495.754997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:32:45.625179951 +0000 UTC m=+1875.414231754" watchObservedRunningTime="2026-09-08 10:32:45.626113892 +0000 UTC m=+1875.415165695" Sep 08 10:32:56.615955 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:32:56.615919 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-869d99c85c-x978q" Sep 08 10:33:03.297406 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:03.297369 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:33:03.795280 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:03.795240 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:33:03.899357 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:03.899323 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-869d99c85c-x978q"] Sep 08 10:33:03.899569 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:03.899543 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-869d99c85c-x978q" podUID="d49494c1-dc53-44ad-a095-2bca054df5e1" containerName="manager" containerID="cri-o://49c28c88956fcc7007340dd86f6a2346042952970f9d188e091d60850067c230" gracePeriod=30 Sep 08 10:33:04.142735 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.142710 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-869d99c85c-x978q" Sep 08 10:33:04.258174 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.258135 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:33:04.317923 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.317837 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d49494c1-dc53-44ad-a095-2bca054df5e1-metrics-certs\") pod \"d49494c1-dc53-44ad-a095-2bca054df5e1\" (UID: \"d49494c1-dc53-44ad-a095-2bca054df5e1\") " Sep 08 10:33:04.317923 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.317908 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkvzf\" (UniqueName: \"kubernetes.io/projected/d49494c1-dc53-44ad-a095-2bca054df5e1-kube-api-access-jkvzf\") pod \"d49494c1-dc53-44ad-a095-2bca054df5e1\" (UID: \"d49494c1-dc53-44ad-a095-2bca054df5e1\") " Sep 08 10:33:04.318376 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.317980 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d49494c1-dc53-44ad-a095-2bca054df5e1-webhook-cert\") pod \"d49494c1-dc53-44ad-a095-2bca054df5e1\" (UID: \"d49494c1-dc53-44ad-a095-2bca054df5e1\") " Sep 08 10:33:04.319986 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.319960 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49494c1-dc53-44ad-a095-2bca054df5e1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d49494c1-dc53-44ad-a095-2bca054df5e1" (UID: "d49494c1-dc53-44ad-a095-2bca054df5e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:33:04.320097 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.320067 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49494c1-dc53-44ad-a095-2bca054df5e1-kube-api-access-jkvzf" (OuterVolumeSpecName: "kube-api-access-jkvzf") pod "d49494c1-dc53-44ad-a095-2bca054df5e1" (UID: "d49494c1-dc53-44ad-a095-2bca054df5e1"). InnerVolumeSpecName "kube-api-access-jkvzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:33:04.320097 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.320079 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49494c1-dc53-44ad-a095-2bca054df5e1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d49494c1-dc53-44ad-a095-2bca054df5e1" (UID: "d49494c1-dc53-44ad-a095-2bca054df5e1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:33:04.419683 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.419639 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d49494c1-dc53-44ad-a095-2bca054df5e1-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:33:04.419683 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.419674 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkvzf\" (UniqueName: \"kubernetes.io/projected/d49494c1-dc53-44ad-a095-2bca054df5e1-kube-api-access-jkvzf\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:33:04.419683 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.419687 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d49494c1-dc53-44ad-a095-2bca054df5e1-webhook-cert\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:33:04.718395 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.718360 2576 generic.go:356] "Generic (PLEG): container finished" podID="d49494c1-dc53-44ad-a095-2bca054df5e1" containerID="49c28c88956fcc7007340dd86f6a2346042952970f9d188e091d60850067c230" exitCode=0 Sep 08 10:33:04.718596 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.718413 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-869d99c85c-x978q" Sep 08 10:33:04.718596 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.718446 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-869d99c85c-x978q" event={"ID":"d49494c1-dc53-44ad-a095-2bca054df5e1","Type":"ContainerDied","Data":"49c28c88956fcc7007340dd86f6a2346042952970f9d188e091d60850067c230"} Sep 08 10:33:04.718596 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.718506 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-869d99c85c-x978q" event={"ID":"d49494c1-dc53-44ad-a095-2bca054df5e1","Type":"ContainerDied","Data":"4d00d261e03654506e2f18ee1791fb645deb2ef6a5de388d45a18813e733a0e5"} Sep 08 10:33:04.718596 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.718538 2576 scope.go:117] "RemoveContainer" containerID="49c28c88956fcc7007340dd86f6a2346042952970f9d188e091d60850067c230" Sep 08 10:33:04.728216 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.728196 2576 scope.go:117] "RemoveContainer" containerID="49c28c88956fcc7007340dd86f6a2346042952970f9d188e091d60850067c230" Sep 08 10:33:04.728545 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:33:04.728525 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c28c88956fcc7007340dd86f6a2346042952970f9d188e091d60850067c230\": container with ID starting with 49c28c88956fcc7007340dd86f6a2346042952970f9d188e091d60850067c230 not found: ID does not exist" containerID="49c28c88956fcc7007340dd86f6a2346042952970f9d188e091d60850067c230" Sep 08 10:33:04.728612 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.728562 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c28c88956fcc7007340dd86f6a2346042952970f9d188e091d60850067c230"} err="failed to get container status \"49c28c88956fcc7007340dd86f6a2346042952970f9d188e091d60850067c230\": rpc error: code = NotFound desc = could not find container \"49c28c88956fcc7007340dd86f6a2346042952970f9d188e091d60850067c230\": container with ID starting with 49c28c88956fcc7007340dd86f6a2346042952970f9d188e091d60850067c230 not found: ID does not exist" Sep 08 10:33:04.741306 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.741283 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-869d99c85c-x978q"] Sep 08 10:33:04.743463 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:04.743444 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-869d99c85c-x978q"] Sep 08 10:33:05.606066 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.606032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:33:05.702316 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.702279 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-869d99c85c-g5wj4"] Sep 08 10:33:05.702685 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.702671 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49494c1-dc53-44ad-a095-2bca054df5e1" containerName="manager" Sep 08 10:33:05.702738 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.702687 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49494c1-dc53-44ad-a095-2bca054df5e1" containerName="manager" Sep 08 10:33:05.702775 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.702760 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="308881df-e633-4279-913c-6d19d1aa9274" containerName="istio-proxy" Sep 08 10:33:05.702775 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.702768 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49494c1-dc53-44ad-a095-2bca054df5e1" containerName="manager" Sep 08 10:33:05.702847 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.702775 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="26271269-74fa-4615-bb71-085a510fe223" containerName="istio-proxy" Sep 08 10:33:05.706919 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.706898 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-869d99c85c-g5wj4" Sep 08 10:33:05.715890 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.715257 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nf7fl\"" Sep 08 10:33:05.715890 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.715374 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 10:33:05.715890 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.715676 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 10:33:05.715890 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.715769 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 10:33:05.719053 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.719032 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-869d99c85c-g5wj4"] Sep 08 10:33:05.729288 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.729264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e14a260-f1c9-4605-a6ed-2fa0caabf045-metrics-certs\") pod \"maas-controller-869d99c85c-g5wj4\" (UID: \"3e14a260-f1c9-4605-a6ed-2fa0caabf045\") " pod="opendatahub/maas-controller-869d99c85c-g5wj4" Sep 08 10:33:05.729400 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.729296 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e14a260-f1c9-4605-a6ed-2fa0caabf045-webhook-cert\") pod \"maas-controller-869d99c85c-g5wj4\" (UID: \"3e14a260-f1c9-4605-a6ed-2fa0caabf045\") " pod="opendatahub/maas-controller-869d99c85c-g5wj4" Sep 08 10:33:05.729400 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.729337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fvt6\" (UniqueName: \"kubernetes.io/projected/3e14a260-f1c9-4605-a6ed-2fa0caabf045-kube-api-access-6fvt6\") pod \"maas-controller-869d99c85c-g5wj4\" (UID: \"3e14a260-f1c9-4605-a6ed-2fa0caabf045\") " pod="opendatahub/maas-controller-869d99c85c-g5wj4" Sep 08 10:33:05.830741 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.830703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e14a260-f1c9-4605-a6ed-2fa0caabf045-metrics-certs\") pod \"maas-controller-869d99c85c-g5wj4\" (UID: \"3e14a260-f1c9-4605-a6ed-2fa0caabf045\") " pod="opendatahub/maas-controller-869d99c85c-g5wj4" Sep 08 10:33:05.830741 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.830745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e14a260-f1c9-4605-a6ed-2fa0caabf045-webhook-cert\") pod \"maas-controller-869d99c85c-g5wj4\" (UID: \"3e14a260-f1c9-4605-a6ed-2fa0caabf045\") " pod="opendatahub/maas-controller-869d99c85c-g5wj4" Sep 08 10:33:05.830973 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.830809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fvt6\" (UniqueName: \"kubernetes.io/projected/3e14a260-f1c9-4605-a6ed-2fa0caabf045-kube-api-access-6fvt6\") pod \"maas-controller-869d99c85c-g5wj4\" (UID: \"3e14a260-f1c9-4605-a6ed-2fa0caabf045\") " pod="opendatahub/maas-controller-869d99c85c-g5wj4" Sep 08 10:33:05.833248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.833217 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e14a260-f1c9-4605-a6ed-2fa0caabf045-metrics-certs\") pod \"maas-controller-869d99c85c-g5wj4\" (UID: \"3e14a260-f1c9-4605-a6ed-2fa0caabf045\") " pod="opendatahub/maas-controller-869d99c85c-g5wj4" Sep 08 10:33:05.833359 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.833345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e14a260-f1c9-4605-a6ed-2fa0caabf045-webhook-cert\") pod \"maas-controller-869d99c85c-g5wj4\" (UID: \"3e14a260-f1c9-4605-a6ed-2fa0caabf045\") " pod="opendatahub/maas-controller-869d99c85c-g5wj4" Sep 08 10:33:05.839757 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:05.839730 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fvt6\" (UniqueName: \"kubernetes.io/projected/3e14a260-f1c9-4605-a6ed-2fa0caabf045-kube-api-access-6fvt6\") pod \"maas-controller-869d99c85c-g5wj4\" (UID: \"3e14a260-f1c9-4605-a6ed-2fa0caabf045\") " pod="opendatahub/maas-controller-869d99c85c-g5wj4" Sep 08 10:33:06.025606 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:06.025564 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-869d99c85c-g5wj4" Sep 08 10:33:06.301078 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:06.301049 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-869d99c85c-g5wj4"] Sep 08 10:33:06.305900 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:33:06.305859 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e14a260_f1c9_4605_a6ed_2fa0caabf045.slice/crio-bbd95fbdcaf927f15063b76b8ad1f7fc6e4242ae60b5f9d899b69943ea3ee502 WatchSource:0}: Error finding container bbd95fbdcaf927f15063b76b8ad1f7fc6e4242ae60b5f9d899b69943ea3ee502: Status 404 returned error can't find the container with id bbd95fbdcaf927f15063b76b8ad1f7fc6e4242ae60b5f9d899b69943ea3ee502 Sep 08 10:33:06.677585 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:06.677557 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d49494c1-dc53-44ad-a095-2bca054df5e1" path="/var/lib/kubelet/pods/d49494c1-dc53-44ad-a095-2bca054df5e1/volumes" Sep 08 10:33:06.734861 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:06.734830 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-869d99c85c-g5wj4" event={"ID":"3e14a260-f1c9-4605-a6ed-2fa0caabf045","Type":"ContainerStarted","Data":"56fcc49316e82a844bdb58ea984019bbb89015171853e43d40e7cea86e6a53ae"} Sep 08 10:33:06.735043 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:06.734899 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-869d99c85c-g5wj4" event={"ID":"3e14a260-f1c9-4605-a6ed-2fa0caabf045","Type":"ContainerStarted","Data":"bbd95fbdcaf927f15063b76b8ad1f7fc6e4242ae60b5f9d899b69943ea3ee502"} Sep 08 10:33:06.735043 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:06.735027 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-869d99c85c-g5wj4" Sep 08 10:33:06.752918 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:06.752859 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-869d99c85c-g5wj4" podStartSLOduration=1.430061215 podStartE2EDuration="1.752844655s" totalImagesPullingTime="322.78344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:33:06.750640733 +0000 UTC m=+1896.539692528" watchObservedRunningTime="2026-09-08 10:33:06.752844655 +0000 UTC m=+1896.541896459" Sep 08 10:33:17.743686 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:17.743654 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-869d99c85c-g5wj4" Sep 08 10:33:29.452101 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:29.452068 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:33:36.541526 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:36.541489 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:33:44.046843 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:44.046810 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:33:52.550328 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:52.550290 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:33:59.758635 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:33:59.758576 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:34:01.100752 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:01.100702 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:34:02.428556 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:02.428524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:34:02.488286 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:02.488259 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:34:13.265807 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:13.265767 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:34:20.349605 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:20.349567 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:34:24.969113 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:24.969035 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd"] Sep 08 10:34:24.969629 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:24.969218 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" podUID="53720794-3846-4a9f-8659-9498a91d7866" containerName="manager" containerID="cri-o://fccf1e181b67b7a81fc668edfe3b66460e129f3723e455b7a9f3182bcfa97928" gracePeriod=10 Sep 08 10:34:27.425172 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:27.425146 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" Sep 08 10:34:27.562923 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:27.562845 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjbwr\" (UniqueName: \"kubernetes.io/projected/53720794-3846-4a9f-8659-9498a91d7866-kube-api-access-cjbwr\") pod \"53720794-3846-4a9f-8659-9498a91d7866\" (UID: \"53720794-3846-4a9f-8659-9498a91d7866\") " Sep 08 10:34:27.563064 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:27.562972 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53720794-3846-4a9f-8659-9498a91d7866-extensions-socket-volume\") pod \"53720794-3846-4a9f-8659-9498a91d7866\" (UID: \"53720794-3846-4a9f-8659-9498a91d7866\") " Sep 08 10:34:27.563323 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:27.563300 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53720794-3846-4a9f-8659-9498a91d7866-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "53720794-3846-4a9f-8659-9498a91d7866" (UID: "53720794-3846-4a9f-8659-9498a91d7866"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 10:34:27.564791 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:27.564770 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53720794-3846-4a9f-8659-9498a91d7866-kube-api-access-cjbwr" (OuterVolumeSpecName: "kube-api-access-cjbwr") pod "53720794-3846-4a9f-8659-9498a91d7866" (UID: "53720794-3846-4a9f-8659-9498a91d7866"). InnerVolumeSpecName "kube-api-access-cjbwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:34:27.664350 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:27.664316 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53720794-3846-4a9f-8659-9498a91d7866-extensions-socket-volume\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:34:27.664350 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:27.664350 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjbwr\" (UniqueName: \"kubernetes.io/projected/53720794-3846-4a9f-8659-9498a91d7866-kube-api-access-cjbwr\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:34:28.190590 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:28.190553 2576 generic.go:356] "Generic (PLEG): container finished" podID="53720794-3846-4a9f-8659-9498a91d7866" containerID="fccf1e181b67b7a81fc668edfe3b66460e129f3723e455b7a9f3182bcfa97928" exitCode=0 Sep 08 10:34:28.190778 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:28.190635 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" Sep 08 10:34:28.190778 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:28.190643 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" event={"ID":"53720794-3846-4a9f-8659-9498a91d7866","Type":"ContainerDied","Data":"fccf1e181b67b7a81fc668edfe3b66460e129f3723e455b7a9f3182bcfa97928"} Sep 08 10:34:28.190778 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:28.190723 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd" event={"ID":"53720794-3846-4a9f-8659-9498a91d7866","Type":"ContainerDied","Data":"598dc8e772f33415526d53a2d5625d3b704e2e06c15663755d348a071dbae8d5"} Sep 08 10:34:28.190778 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:28.190744 2576 scope.go:117] "RemoveContainer" containerID="fccf1e181b67b7a81fc668edfe3b66460e129f3723e455b7a9f3182bcfa97928" Sep 08 10:34:28.201240 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:28.201221 2576 scope.go:117] "RemoveContainer" containerID="fccf1e181b67b7a81fc668edfe3b66460e129f3723e455b7a9f3182bcfa97928" Sep 08 10:34:28.201555 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:34:28.201534 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fccf1e181b67b7a81fc668edfe3b66460e129f3723e455b7a9f3182bcfa97928\": container with ID starting with fccf1e181b67b7a81fc668edfe3b66460e129f3723e455b7a9f3182bcfa97928 not found: ID does not exist" containerID="fccf1e181b67b7a81fc668edfe3b66460e129f3723e455b7a9f3182bcfa97928" Sep 08 10:34:28.201602 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:28.201565 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fccf1e181b67b7a81fc668edfe3b66460e129f3723e455b7a9f3182bcfa97928"} err="failed to get container status \"fccf1e181b67b7a81fc668edfe3b66460e129f3723e455b7a9f3182bcfa97928\": rpc error: code = NotFound desc = could not find container \"fccf1e181b67b7a81fc668edfe3b66460e129f3723e455b7a9f3182bcfa97928\": container with ID starting with fccf1e181b67b7a81fc668edfe3b66460e129f3723e455b7a9f3182bcfa97928 not found: ID does not exist" Sep 08 10:34:28.212190 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:28.212165 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd"] Sep 08 10:34:28.218006 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:28.217983 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-twmwd"] Sep 08 10:34:28.677209 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:28.677179 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53720794-3846-4a9f-8659-9498a91d7866" path="/var/lib/kubelet/pods/53720794-3846-4a9f-8659-9498a91d7866/volumes" Sep 08 10:34:31.394628 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:34:31.394581 2576 scope.go:117] "RemoveContainer" containerID="f7f96bfe2cfc4ded38c612e3ca157dbe4f01cc3f1a8639c63f711b3f66ceccc5" Sep 08 10:35:25.674776 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:25.674740 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk"] Sep 08 10:35:25.675171 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:25.675107 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53720794-3846-4a9f-8659-9498a91d7866" containerName="manager" Sep 08 10:35:25.675171 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:25.675117 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="53720794-3846-4a9f-8659-9498a91d7866" containerName="manager" Sep 08 10:35:25.675242 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:25.675192 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="53720794-3846-4a9f-8659-9498a91d7866" containerName="manager" Sep 08 10:35:25.678175 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:25.678159 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk" Sep 08 10:35:25.680528 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:25.680507 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pz647\"" Sep 08 10:35:25.691795 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:25.691772 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk"] Sep 08 10:35:25.761757 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:25.761731 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42sbg\" (UniqueName: \"kubernetes.io/projected/2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b-kube-api-access-42sbg\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-q82bk\" (UID: \"2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk" Sep 08 10:35:25.761917 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:25.761778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-q82bk\" (UID: \"2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk" Sep 08 10:35:25.862672 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:25.862648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42sbg\" (UniqueName: \"kubernetes.io/projected/2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b-kube-api-access-42sbg\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-q82bk\" (UID: \"2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk" Sep 08 10:35:25.862810 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:25.862691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-q82bk\" (UID: \"2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk" Sep 08 10:35:25.863059 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:25.863041 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-q82bk\" (UID: \"2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk" Sep 08 10:35:25.875169 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:25.875141 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42sbg\" (UniqueName: \"kubernetes.io/projected/2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b-kube-api-access-42sbg\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-q82bk\" (UID: \"2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk" Sep 08 10:35:25.988236 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:25.988180 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk" Sep 08 10:35:26.270308 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:26.270284 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk"] Sep 08 10:35:26.271713 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:35:26.271684 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7be960_ac4a_40c2_9c4d_7cabbc6e2c6b.slice/crio-9df1a245fed4daf7bbbe3c3dd359f150c8d161da5fb11ad407eb711f86248f77 WatchSource:0}: Error finding container 9df1a245fed4daf7bbbe3c3dd359f150c8d161da5fb11ad407eb711f86248f77: Status 404 returned error can't find the container with id 9df1a245fed4daf7bbbe3c3dd359f150c8d161da5fb11ad407eb711f86248f77 Sep 08 10:35:26.507368 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:26.507336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk" event={"ID":"2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b","Type":"ContainerStarted","Data":"7d3ba7b4d57b9e806289163e0e99e7444847c2f56615c4f6c96422ca197fd511"} Sep 08 10:35:26.507536 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:26.507383 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk" event={"ID":"2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b","Type":"ContainerStarted","Data":"9df1a245fed4daf7bbbe3c3dd359f150c8d161da5fb11ad407eb711f86248f77"} Sep 08 10:35:26.507536 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:26.507428 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk" Sep 08 10:35:26.528841 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:26.528753 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk" podStartSLOduration=1.5287378459999998 podStartE2EDuration="1.528737846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:35:26.525650253 +0000 UTC m=+2036.314702048" watchObservedRunningTime="2026-09-08 10:35:26.528737846 +0000 UTC m=+2036.317789648" Sep 08 10:35:37.515856 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:37.515826 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-q82bk" Sep 08 10:35:44.754018 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:44.753982 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:35:53.150879 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:53.150793 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:35:59.149567 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:35:59.149530 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:36:15.219248 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:36:15.219178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:36:17.558758 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:36:17.558717 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:36:18.330948 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:36:18.330913 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:36:31.976713 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:36:31.976678 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:36:34.745427 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:36:34.745392 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:37:00.697348 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:00.697299 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:37:01.766776 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:01.766746 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:37:03.756117 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:03.756077 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:37:11.155897 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:11.155855 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:37:18.449816 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:18.449778 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:37:18.855256 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:18.855177 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj"] Sep 08 10:37:18.859535 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:18.859510 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:18.861750 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:18.861728 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w6wth\"" Sep 08 10:37:18.867858 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:18.867793 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj"] Sep 08 10:37:18.985840 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:18.985804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c10235-aac9-42ac-be1f-692834640bd1-metrics-certs\") pod \"maas-api-7cdb685b44-4jrtj\" (UID: \"d1c10235-aac9-42ac-be1f-692834640bd1\") " pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:18.985840 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:18.985844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c10235-aac9-42ac-be1f-692834640bd1-maas-api-tls\") pod \"maas-api-7cdb685b44-4jrtj\" (UID: \"d1c10235-aac9-42ac-be1f-692834640bd1\") " pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:18.986052 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:18.985894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c10235-aac9-42ac-be1f-692834640bd1-openshift-service-ca\") pod \"maas-api-7cdb685b44-4jrtj\" (UID: \"d1c10235-aac9-42ac-be1f-692834640bd1\") " pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:18.986052 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:18.985927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn4rv\" (UniqueName: \"kubernetes.io/projected/d1c10235-aac9-42ac-be1f-692834640bd1-kube-api-access-qn4rv\") pod \"maas-api-7cdb685b44-4jrtj\" (UID: \"d1c10235-aac9-42ac-be1f-692834640bd1\") " pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:19.087014 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:19.086977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c10235-aac9-42ac-be1f-692834640bd1-metrics-certs\") pod \"maas-api-7cdb685b44-4jrtj\" (UID: \"d1c10235-aac9-42ac-be1f-692834640bd1\") " pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:19.087014 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:19.087016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c10235-aac9-42ac-be1f-692834640bd1-maas-api-tls\") pod \"maas-api-7cdb685b44-4jrtj\" (UID: \"d1c10235-aac9-42ac-be1f-692834640bd1\") " pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:19.087283 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:19.087058 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c10235-aac9-42ac-be1f-692834640bd1-openshift-service-ca\") pod \"maas-api-7cdb685b44-4jrtj\" (UID: \"d1c10235-aac9-42ac-be1f-692834640bd1\") " pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:19.087283 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:19.087079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn4rv\" (UniqueName: \"kubernetes.io/projected/d1c10235-aac9-42ac-be1f-692834640bd1-kube-api-access-qn4rv\") pod \"maas-api-7cdb685b44-4jrtj\" (UID: \"d1c10235-aac9-42ac-be1f-692834640bd1\") " pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:19.087831 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:19.087811 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c10235-aac9-42ac-be1f-692834640bd1-openshift-service-ca\") pod \"maas-api-7cdb685b44-4jrtj\" (UID: \"d1c10235-aac9-42ac-be1f-692834640bd1\") " pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:19.089603 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:19.089578 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c10235-aac9-42ac-be1f-692834640bd1-maas-api-tls\") pod \"maas-api-7cdb685b44-4jrtj\" (UID: \"d1c10235-aac9-42ac-be1f-692834640bd1\") " pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:19.089832 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:19.089695 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c10235-aac9-42ac-be1f-692834640bd1-metrics-certs\") pod \"maas-api-7cdb685b44-4jrtj\" (UID: \"d1c10235-aac9-42ac-be1f-692834640bd1\") " pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:19.095114 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:19.095089 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn4rv\" (UniqueName: \"kubernetes.io/projected/d1c10235-aac9-42ac-be1f-692834640bd1-kube-api-access-qn4rv\") pod \"maas-api-7cdb685b44-4jrtj\" (UID: \"d1c10235-aac9-42ac-be1f-692834640bd1\") " pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:19.172267 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:19.172233 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:19.448540 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:19.448514 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj"] Sep 08 10:37:19.450010 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:37:19.449984 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c10235_aac9_42ac_be1f_692834640bd1.slice/crio-334f067d05cbf9dffcc54d574585f2dd84c12d4aa0389347f7117616860a67ee WatchSource:0}: Error finding container 334f067d05cbf9dffcc54d574585f2dd84c12d4aa0389347f7117616860a67ee: Status 404 returned error can't find the container with id 334f067d05cbf9dffcc54d574585f2dd84c12d4aa0389347f7117616860a67ee Sep 08 10:37:20.126840 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:20.126790 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" event={"ID":"d1c10235-aac9-42ac-be1f-692834640bd1","Type":"ContainerStarted","Data":"4b8b57f39bad418cfb066507a22751174718b0a50c4ad1a1fe484769091f54e6"} Sep 08 10:37:20.127033 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:20.126863 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" event={"ID":"d1c10235-aac9-42ac-be1f-692834640bd1","Type":"ContainerStarted","Data":"334f067d05cbf9dffcc54d574585f2dd84c12d4aa0389347f7117616860a67ee"} Sep 08 10:37:20.127033 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:20.126894 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:20.144956 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:20.144905 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" podStartSLOduration=1.687941574 podStartE2EDuration="2.144888818s" totalImagesPullingTime="456.947244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:37:20.142776212 +0000 UTC m=+2149.931828034" watchObservedRunningTime="2026-09-08 10:37:20.144888818 +0000 UTC m=+2149.933940673" Sep 08 10:37:26.137598 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:26.137518 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7cdb685b44-4jrtj" Sep 08 10:37:27.849927 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:27.849891 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:37:33.875235 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:33.875183 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5"] Sep 08 10:37:33.875757 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:33.875427 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" podUID="743cdc55-c328-4650-b2f5-f0bdff2c89ee" containerName="maas-api" containerID="cri-o://eff8c101ae0a4304b0bd2c7e3d9e7fd8d6c380b211ebe5c87008a4baa68234c4" gracePeriod=30 Sep 08 10:37:35.063290 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.063261 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:37:35.125866 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.125843 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:37:35.210261 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.210228 2576 generic.go:356] "Generic (PLEG): container finished" podID="743cdc55-c328-4650-b2f5-f0bdff2c89ee" containerID="eff8c101ae0a4304b0bd2c7e3d9e7fd8d6c380b211ebe5c87008a4baa68234c4" exitCode=0 Sep 08 10:37:35.210426 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.210278 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" Sep 08 10:37:35.210426 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.210309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" event={"ID":"743cdc55-c328-4650-b2f5-f0bdff2c89ee","Type":"ContainerDied","Data":"eff8c101ae0a4304b0bd2c7e3d9e7fd8d6c380b211ebe5c87008a4baa68234c4"} Sep 08 10:37:35.210426 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.210372 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5" event={"ID":"743cdc55-c328-4650-b2f5-f0bdff2c89ee","Type":"ContainerDied","Data":"f59350fd0118daf0a1926c1916c9a75727f199cb8b4f941879789e969fbf54f0"} Sep 08 10:37:35.210426 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.210398 2576 scope.go:117] "RemoveContainer" containerID="eff8c101ae0a4304b0bd2c7e3d9e7fd8d6c380b211ebe5c87008a4baa68234c4" Sep 08 10:37:35.219393 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.219375 2576 scope.go:117] "RemoveContainer" containerID="eff8c101ae0a4304b0bd2c7e3d9e7fd8d6c380b211ebe5c87008a4baa68234c4" Sep 08 10:37:35.219712 ip-10-0-132-206 kubenswrapper[2576]: E0908 10:37:35.219690 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff8c101ae0a4304b0bd2c7e3d9e7fd8d6c380b211ebe5c87008a4baa68234c4\": container with ID starting with eff8c101ae0a4304b0bd2c7e3d9e7fd8d6c380b211ebe5c87008a4baa68234c4 not found: ID does not exist" containerID="eff8c101ae0a4304b0bd2c7e3d9e7fd8d6c380b211ebe5c87008a4baa68234c4" Sep 08 10:37:35.219794 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.219718 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff8c101ae0a4304b0bd2c7e3d9e7fd8d6c380b211ebe5c87008a4baa68234c4"} err="failed to get container status \"eff8c101ae0a4304b0bd2c7e3d9e7fd8d6c380b211ebe5c87008a4baa68234c4\": rpc error: code = NotFound desc = could not find container \"eff8c101ae0a4304b0bd2c7e3d9e7fd8d6c380b211ebe5c87008a4baa68234c4\": container with ID starting with eff8c101ae0a4304b0bd2c7e3d9e7fd8d6c380b211ebe5c87008a4baa68234c4 not found: ID does not exist" Sep 08 10:37:35.228998 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.228978 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/743cdc55-c328-4650-b2f5-f0bdff2c89ee-maas-api-tls\") pod \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " Sep 08 10:37:35.229086 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.229019 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/743cdc55-c328-4650-b2f5-f0bdff2c89ee-openshift-service-ca\") pod \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " Sep 08 10:37:35.229086 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.229065 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/743cdc55-c328-4650-b2f5-f0bdff2c89ee-metrics-certs\") pod \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " Sep 08 10:37:35.229171 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.229140 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hksr8\" (UniqueName: \"kubernetes.io/projected/743cdc55-c328-4650-b2f5-f0bdff2c89ee-kube-api-access-hksr8\") pod \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\" (UID: \"743cdc55-c328-4650-b2f5-f0bdff2c89ee\") " Sep 08 10:37:35.229414 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.229389 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743cdc55-c328-4650-b2f5-f0bdff2c89ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "743cdc55-c328-4650-b2f5-f0bdff2c89ee" (UID: "743cdc55-c328-4650-b2f5-f0bdff2c89ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 10:37:35.231065 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.231040 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/743cdc55-c328-4650-b2f5-f0bdff2c89ee-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "743cdc55-c328-4650-b2f5-f0bdff2c89ee" (UID: "743cdc55-c328-4650-b2f5-f0bdff2c89ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:37:35.231152 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.231102 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/743cdc55-c328-4650-b2f5-f0bdff2c89ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "743cdc55-c328-4650-b2f5-f0bdff2c89ee" (UID: "743cdc55-c328-4650-b2f5-f0bdff2c89ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 10:37:35.231152 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.231104 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743cdc55-c328-4650-b2f5-f0bdff2c89ee-kube-api-access-hksr8" (OuterVolumeSpecName: "kube-api-access-hksr8") pod "743cdc55-c328-4650-b2f5-f0bdff2c89ee" (UID: "743cdc55-c328-4650-b2f5-f0bdff2c89ee"). InnerVolumeSpecName "kube-api-access-hksr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 10:37:35.330799 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.330769 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/743cdc55-c328-4650-b2f5-f0bdff2c89ee-openshift-service-ca\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:37:35.330799 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.330797 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/743cdc55-c328-4650-b2f5-f0bdff2c89ee-metrics-certs\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:37:35.330799 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.330806 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hksr8\" (UniqueName: \"kubernetes.io/projected/743cdc55-c328-4650-b2f5-f0bdff2c89ee-kube-api-access-hksr8\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:37:35.331013 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.330814 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/743cdc55-c328-4650-b2f5-f0bdff2c89ee-maas-api-tls\") on node \"ip-10-0-132-206.ec2.internal\" DevicePath \"\"" Sep 08 10:37:35.536314 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.536281 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5"] Sep 08 10:37:35.538702 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:35.538681 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-787f6cbf7b-kvfw5"] Sep 08 10:37:36.678258 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:36.678223 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="743cdc55-c328-4650-b2f5-f0bdff2c89ee" path="/var/lib/kubelet/pods/743cdc55-c328-4650-b2f5-f0bdff2c89ee/volumes" Sep 08 10:37:40.246150 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:40.246118 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:37:49.349490 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:49.349454 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-s5tht"] Sep 08 10:37:56.418069 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:56.418038 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-25nft_250f37c3-df6f-44b0-b16c-6a14c05d3bd8/manager/0.log" Sep 08 10:37:56.535749 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:56.535706 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wch2r_1ab54ac7-8c16-4fbd-b68f-02884684490e/manager/0.log" Sep 08 10:37:56.654720 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:56.654690 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-869d99c85c-g5wj4_3e14a260-f1c9-4605-a6ed-2fa0caabf045/manager/0.log" Sep 08 10:37:56.759110 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:56.759031 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-84rwh_563497e2-7695-4b69-89a8-e9f44721c58f/server/0.log" Sep 08 10:37:56.869290 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:56.869260 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-n9qmd_2e92279a-8448-4bcf-9b75-4b80f73c3f9d/manager/0.log" Sep 08 10:37:57.248217 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:57.248192 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b6d9c7b-xvsph_17f6623a-31b9-4352-b713-ecbb082847f6/copy-manifests/0.log" Sep 08 10:37:57.254956 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:57.254937 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b6d9c7b-xvsph_17f6623a-31b9-4352-b713-ecbb082847f6/manager/0.log" Sep 08 10:37:57.598359 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:57.598280 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7cdb685b44-4jrtj_d1c10235-aac9-42ac-be1f-692834640bd1/maas-api/0.log" Sep 08 10:37:58.995819 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:58.995788 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-bllss_4110d091-f0cb-45c6-8350-0ed142047c6c/manager/0.log" Sep 08 10:37:59.111797 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:59.111769 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-mpq95_9646f179-ace5-4b54-a40d-fbf644218d46/manager/0.log" Sep 08 10:37:59.223269 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:59.223240 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-b2czx_b43829ee-9034-42b6-92fe-9b4f7f66392b/kuadrant-console-plugin/0.log" Sep 08 10:37:59.347255 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:59.347170 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-q82bk_2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b/manager/0.log" Sep 08 10:37:59.471214 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:37:59.471183 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-s5tht_a27e588e-b522-447f-a566-2c8320de409e/limitador/0.log" Sep 08 10:38:00.331705 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:00.331665 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c9fc4cdc-v9jpr_53096f12-27bc-4599-8c5c-a31c72bc3eee/kube-auth-proxy/0.log" Sep 08 10:38:00.447235 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:00.447207 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2m56t_48567896-0f21-404c-863a-1ece4631e205/istio-proxy/1.log" Sep 08 10:38:00.571636 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:00.571596 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-qljfd_e6596df5-5301-45a5-9116-dc2a4268bc80/payload-pre-processing/0.log" Sep 08 10:38:01.177522 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:01.177491 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-8jrfr_46f3579d-8372-46cf-8cb3-466410fe4552/main/0.log" Sep 08 10:38:07.290939 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.290907 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:38:07.306076 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.306044 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q6fc7/must-gather-jb6jm"] Sep 08 10:38:07.306433 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.306421 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="743cdc55-c328-4650-b2f5-f0bdff2c89ee" containerName="maas-api" Sep 08 10:38:07.306433 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.306433 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="743cdc55-c328-4650-b2f5-f0bdff2c89ee" containerName="maas-api" Sep 08 10:38:07.306577 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.306491 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="743cdc55-c328-4650-b2f5-f0bdff2c89ee" containerName="maas-api" Sep 08 10:38:07.309468 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.309451 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q6fc7/must-gather-jb6jm" Sep 08 10:38:07.312402 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.312383 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q6fc7\"/\"kube-root-ca.crt\"" Sep 08 10:38:07.312402 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.312392 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q6fc7\"/\"default-dockercfg-hchl5\"" Sep 08 10:38:07.312691 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.312675 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q6fc7\"/\"openshift-service-ca.crt\"" Sep 08 10:38:07.322270 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.322239 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q6fc7/must-gather-jb6jm"] Sep 08 10:38:07.407689 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.407651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b64950c0-9150-4b1b-be52-70a73c96b43a-must-gather-output\") pod \"must-gather-jb6jm\" (UID: \"b64950c0-9150-4b1b-be52-70a73c96b43a\") " pod="openshift-must-gather-q6fc7/must-gather-jb6jm" Sep 08 10:38:07.407689 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.407697 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kzmg\" (UniqueName: \"kubernetes.io/projected/b64950c0-9150-4b1b-be52-70a73c96b43a-kube-api-access-9kzmg\") pod \"must-gather-jb6jm\" (UID: \"b64950c0-9150-4b1b-be52-70a73c96b43a\") " pod="openshift-must-gather-q6fc7/must-gather-jb6jm" Sep 08 10:38:07.508938 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.508901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b64950c0-9150-4b1b-be52-70a73c96b43a-must-gather-output\") pod \"must-gather-jb6jm\" (UID: \"b64950c0-9150-4b1b-be52-70a73c96b43a\") " pod="openshift-must-gather-q6fc7/must-gather-jb6jm" Sep 08 10:38:07.508938 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.508946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kzmg\" (UniqueName: \"kubernetes.io/projected/b64950c0-9150-4b1b-be52-70a73c96b43a-kube-api-access-9kzmg\") pod \"must-gather-jb6jm\" (UID: \"b64950c0-9150-4b1b-be52-70a73c96b43a\") " pod="openshift-must-gather-q6fc7/must-gather-jb6jm" Sep 08 10:38:07.509260 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.509238 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b64950c0-9150-4b1b-be52-70a73c96b43a-must-gather-output\") pod \"must-gather-jb6jm\" (UID: \"b64950c0-9150-4b1b-be52-70a73c96b43a\") " pod="openshift-must-gather-q6fc7/must-gather-jb6jm" Sep 08 10:38:07.517244 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.517217 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kzmg\" (UniqueName: \"kubernetes.io/projected/b64950c0-9150-4b1b-be52-70a73c96b43a-kube-api-access-9kzmg\") pod \"must-gather-jb6jm\" (UID: \"b64950c0-9150-4b1b-be52-70a73c96b43a\") " pod="openshift-must-gather-q6fc7/must-gather-jb6jm" Sep 08 10:38:07.622753 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.622651 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q6fc7/must-gather-jb6jm" Sep 08 10:38:07.901273 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.901248 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q6fc7/must-gather-jb6jm"] Sep 08 10:38:07.903505 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:38:07.903471 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb64950c0_9150_4b1b_be52_70a73c96b43a.slice/crio-ffdc8b95e758269e5bd849ef7a164b3d67156c89b9204ef9b9718d8f55a410c2 WatchSource:0}: Error finding container ffdc8b95e758269e5bd849ef7a164b3d67156c89b9204ef9b9718d8f55a410c2: Status 404 returned error can't find the container with id ffdc8b95e758269e5bd849ef7a164b3d67156c89b9204ef9b9718d8f55a410c2 Sep 08 10:38:07.905372 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:07.905355 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 10:38:08.396474 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:08.396441 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q6fc7/must-gather-jb6jm" event={"ID":"b64950c0-9150-4b1b-be52-70a73c96b43a","Type":"ContainerStarted","Data":"ffdc8b95e758269e5bd849ef7a164b3d67156c89b9204ef9b9718d8f55a410c2"} Sep 08 10:38:08.707457 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:08.707357 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:38:09.406884 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:09.406714 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q6fc7/must-gather-jb6jm" event={"ID":"b64950c0-9150-4b1b-be52-70a73c96b43a","Type":"ContainerStarted","Data":"b0477149af70d54928d2171db1138726534d887da6d4afcadb7455358d894840"} Sep 08 10:38:09.406884 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:09.406770 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q6fc7/must-gather-jb6jm" event={"ID":"b64950c0-9150-4b1b-be52-70a73c96b43a","Type":"ContainerStarted","Data":"ab7673c5023270763a1d75556080c0a0f1b4a74441570535d61f97ad64666504"} Sep 08 10:38:09.425531 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:09.425482 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q6fc7/must-gather-jb6jm" podStartSLOduration=1.145287269 podStartE2EDuration="2.425465469s" totalImagesPullingTime="1.2801782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:38:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:38:09.424845927 +0000 UTC m=+2199.213897735" watchObservedRunningTime="2026-09-08 10:38:09.425465469 +0000 UTC m=+2199.214517274" Sep 08 10:38:10.863276 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:10.863243 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ktdpl_2b7ee997-0723-4a47-90b5-805cd7721382/global-pull-secret-syncer/0.log" Sep 08 10:38:10.920177 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:10.920141 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" Sep 08 10:38:11.054675 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:11.054643 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-206.ec2.internal_aee6609005f8620b26930034856bde0d/haproxy/0.log" Sep 08 10:38:15.465926 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:15.465893 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-bllss_4110d091-f0cb-45c6-8350-0ed142047c6c/manager/0.log" Sep 08 10:38:15.502543 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:15.502513 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-mpq95_9646f179-ace5-4b54-a40d-fbf644218d46/manager/0.log" Sep 08 10:38:15.535040 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:15.535013 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-b2czx_b43829ee-9034-42b6-92fe-9b4f7f66392b/kuadrant-console-plugin/0.log" Sep 08 10:38:15.578703 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:15.578676 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-q82bk_2a7be960-ac4a-40c2-9c4d-7cabbc6e2c6b/manager/0.log" Sep 08 10:38:15.603645 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:15.603600 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-s5tht_a27e588e-b522-447f-a566-2c8320de409e/limitador/0.log" Sep 08 10:38:17.286487 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:17.286451 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3d01ee9c-4452-40e8-ba37-40dc870d36b2/alertmanager/0.log" Sep 08 10:38:17.315352 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:17.315317 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3d01ee9c-4452-40e8-ba37-40dc870d36b2/config-reloader/0.log" Sep 08 10:38:17.338071 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:17.338014 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3d01ee9c-4452-40e8-ba37-40dc870d36b2/kube-rbac-proxy-web/0.log" Sep 08 10:38:17.364299 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:17.364275 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3d01ee9c-4452-40e8-ba37-40dc870d36b2/kube-rbac-proxy/0.log" Sep 08 10:38:17.389538 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:17.389499 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3d01ee9c-4452-40e8-ba37-40dc870d36b2/kube-rbac-proxy-metric/0.log" Sep 08 10:38:17.411904 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:17.411880 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3d01ee9c-4452-40e8-ba37-40dc870d36b2/prom-label-proxy/0.log" Sep 08 10:38:17.440979 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:17.440913 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3d01ee9c-4452-40e8-ba37-40dc870d36b2/init-config-reloader/0.log" Sep 08 10:38:17.903546 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:17.903477 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bz65z_e2d18286-9927-4254-9ae7-0d1cd0470ae1/node-exporter/0.log" Sep 08 10:38:17.939529 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:17.939495 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bz65z_e2d18286-9927-4254-9ae7-0d1cd0470ae1/kube-rbac-proxy/0.log" Sep 08 10:38:17.964802 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:17.964764 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bz65z_e2d18286-9927-4254-9ae7-0d1cd0470ae1/init-textfile/0.log" Sep 08 10:38:18.268072 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:18.267977 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-d72f2_01943a4b-bbca-418d-add3-d0169cd20bff/prometheus-operator/0.log" Sep 08 10:38:18.291812 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:18.291776 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-d72f2_01943a4b-bbca-418d-add3-d0169cd20bff/kube-rbac-proxy/0.log" Sep 08 10:38:21.051473 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.051447 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-879476b9f-vc6q4_ffdc38d6-64a0-4164-88a8-459e2c5ba746/console/0.log" Sep 08 10:38:21.094221 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.094184 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-qhcfl_00a89ef4-4cc0-4f8c-b01f-b8dfa24e340a/download-server/0.log" Sep 08 10:38:21.150478 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.150434 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq"] Sep 08 10:38:21.156279 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.156250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.162211 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.162178 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq"] Sep 08 10:38:21.252347 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.252302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d72014e-6077-47b6-950f-f256a6ee155d-lib-modules\") pod \"perf-node-gather-daemonset-lb8gq\" (UID: \"5d72014e-6077-47b6-950f-f256a6ee155d\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.252570 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.252410 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d72014e-6077-47b6-950f-f256a6ee155d-podres\") pod \"perf-node-gather-daemonset-lb8gq\" (UID: \"5d72014e-6077-47b6-950f-f256a6ee155d\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.252570 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.252466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d72014e-6077-47b6-950f-f256a6ee155d-sys\") pod \"perf-node-gather-daemonset-lb8gq\" (UID: \"5d72014e-6077-47b6-950f-f256a6ee155d\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.252570 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.252498 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9lh\" (UniqueName: \"kubernetes.io/projected/5d72014e-6077-47b6-950f-f256a6ee155d-kube-api-access-df9lh\") pod \"perf-node-gather-daemonset-lb8gq\" (UID: \"5d72014e-6077-47b6-950f-f256a6ee155d\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.252570 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.252528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d72014e-6077-47b6-950f-f256a6ee155d-proc\") pod \"perf-node-gather-daemonset-lb8gq\" (UID: \"5d72014e-6077-47b6-950f-f256a6ee155d\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.353238 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.353142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d72014e-6077-47b6-950f-f256a6ee155d-sys\") pod \"perf-node-gather-daemonset-lb8gq\" (UID: \"5d72014e-6077-47b6-950f-f256a6ee155d\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.353238 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.353188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9lh\" (UniqueName: \"kubernetes.io/projected/5d72014e-6077-47b6-950f-f256a6ee155d-kube-api-access-df9lh\") pod \"perf-node-gather-daemonset-lb8gq\" (UID: \"5d72014e-6077-47b6-950f-f256a6ee155d\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.353238 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.353216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d72014e-6077-47b6-950f-f256a6ee155d-proc\") pod \"perf-node-gather-daemonset-lb8gq\" (UID: \"5d72014e-6077-47b6-950f-f256a6ee155d\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.353517 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.353283 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d72014e-6077-47b6-950f-f256a6ee155d-sys\") pod \"perf-node-gather-daemonset-lb8gq\" (UID: \"5d72014e-6077-47b6-950f-f256a6ee155d\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.353517 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.353298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d72014e-6077-47b6-950f-f256a6ee155d-lib-modules\") pod \"perf-node-gather-daemonset-lb8gq\" (UID: \"5d72014e-6077-47b6-950f-f256a6ee155d\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.353517 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.353303 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d72014e-6077-47b6-950f-f256a6ee155d-proc\") pod \"perf-node-gather-daemonset-lb8gq\" (UID: \"5d72014e-6077-47b6-950f-f256a6ee155d\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.353517 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.353375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d72014e-6077-47b6-950f-f256a6ee155d-podres\") pod \"perf-node-gather-daemonset-lb8gq\" (UID: \"5d72014e-6077-47b6-950f-f256a6ee155d\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.353517 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.353424 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d72014e-6077-47b6-950f-f256a6ee155d-lib-modules\") pod \"perf-node-gather-daemonset-lb8gq\" (UID: \"5d72014e-6077-47b6-950f-f256a6ee155d\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.353517 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.353500 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d72014e-6077-47b6-950f-f256a6ee155d-podres\") pod \"perf-node-gather-daemonset-lb8gq\" (UID: \"5d72014e-6077-47b6-950f-f256a6ee155d\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.362412 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.362374 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9lh\" (UniqueName: \"kubernetes.io/projected/5d72014e-6077-47b6-950f-f256a6ee155d-kube-api-access-df9lh\") pod \"perf-node-gather-daemonset-lb8gq\" (UID: \"5d72014e-6077-47b6-950f-f256a6ee155d\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.476870 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.476823 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:21.764364 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:21.764298 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq"] Sep 08 10:38:21.772148 ip-10-0-132-206 kubenswrapper[2576]: W0908 10:38:21.772105 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5d72014e_6077_47b6_950f_f256a6ee155d.slice/crio-b49b50716fd4d97b2c1269381766bb050c1823a94cd41659b181eca2dc8d6ca4 WatchSource:0}: Error finding container b49b50716fd4d97b2c1269381766bb050c1823a94cd41659b181eca2dc8d6ca4: Status 404 returned error can't find the container with id b49b50716fd4d97b2c1269381766bb050c1823a94cd41659b181eca2dc8d6ca4 Sep 08 10:38:22.519757 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:22.519711 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" event={"ID":"5d72014e-6077-47b6-950f-f256a6ee155d","Type":"ContainerStarted","Data":"25eefebc0812faa883e2612a0c1fcf39b83ecde0b93bbdf83eee27e8f9bb83e5"} Sep 08 10:38:22.520229 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:22.519786 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" event={"ID":"5d72014e-6077-47b6-950f-f256a6ee155d","Type":"ContainerStarted","Data":"b49b50716fd4d97b2c1269381766bb050c1823a94cd41659b181eca2dc8d6ca4"} Sep 08 10:38:22.520721 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:22.520690 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:22.540079 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:22.540010 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" podStartSLOduration=1.5399877389999999 podStartE2EDuration="1.539987739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:38:22.536930809 +0000 UTC m=+2212.325982614" watchObservedRunningTime="2026-09-08 10:38:22.539987739 +0000 UTC m=+2212.329039545" Sep 08 10:38:23.130182 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:23.130152 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n64nz_acdded14-6f01-4f17-97ea-7b9f7f1006eb/dns/0.log" Sep 08 10:38:23.153968 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:23.153936 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n64nz_acdded14-6f01-4f17-97ea-7b9f7f1006eb/kube-rbac-proxy/0.log" Sep 08 10:38:23.279226 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:23.279185 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-95fpk_d5114137-2635-4a57-bdae-7866a1b417a2/dns-node-resolver/0.log" Sep 08 10:38:23.857247 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:23.857210 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6666db957f-c2qnq_9f4222e2-59da-4454-bc94-a6fbee924a54/registry/0.log" Sep 08 10:38:23.931359 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:23.931326 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wqx5v_12807ade-0ca9-4bd3-b277-afba07e51a26/node-ca/0.log" Sep 08 10:38:25.224525 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:25.224495 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c9fc4cdc-v9jpr_53096f12-27bc-4599-8c5c-a31c72bc3eee/kube-auth-proxy/0.log" Sep 08 10:38:25.256067 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:25.256037 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2m56t_48567896-0f21-404c-863a-1ece4631e205/istio-proxy/0.log" Sep 08 10:38:25.268175 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:25.268150 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2m56t_48567896-0f21-404c-863a-1ece4631e205/istio-proxy/1.log" Sep 08 10:38:25.312373 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:25.312344 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-qljfd_e6596df5-5301-45a5-9116-dc2a4268bc80/payload-pre-processing/0.log" Sep 08 10:38:26.256425 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:26.256394 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d8gdx_d09fba07-6330-44b8-b463-0a48b3d9e70b/serve-healthcheck-canary/0.log" Sep 08 10:38:26.700682 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:26.700643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9nb_1ed41459-ca2d-4277-a2df-c7879f425183/konnectivity-agent/0.log" limit=1024 Sep 08 10:38:26.799763 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:26.799725 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hqtdm_fb5984d1-f23f-49fd-95a3-aaa11bb607a0/kube-rbac-proxy/0.log" Sep 08 10:38:26.822417 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:26.822393 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hqtdm_fb5984d1-f23f-49fd-95a3-aaa11bb607a0/exporter/0.log" Sep 08 10:38:26.852352 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:26.852323 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hqtdm_fb5984d1-f23f-49fd-95a3-aaa11bb607a0/extractor/0.log" Sep 08 10:38:29.400448 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:29.400422 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-25nft_250f37c3-df6f-44b0-b16c-6a14c05d3bd8/manager/0.log" Sep 08 10:38:29.552973 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:29.552938 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-lb8gq" Sep 08 10:38:29.622565 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:29.622524 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wch2r_1ab54ac7-8c16-4fbd-b68f-02884684490e/manager/0.log" Sep 08 10:38:29.676702 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:29.676605 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-869d99c85c-g5wj4_3e14a260-f1c9-4605-a6ed-2fa0caabf045/manager/0.log" Sep 08 10:38:29.698121 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:29.698083 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-84rwh_563497e2-7695-4b69-89a8-e9f44721c58f/server/0.log" Sep 08 10:38:29.792340 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:29.792309 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-n9qmd_2e92279a-8448-4bcf-9b75-4b80f73c3f9d/manager/0.log" Sep 08 10:38:29.963784 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:29.963688 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b6d9c7b-xvsph_17f6623a-31b9-4352-b713-ecbb082847f6/manager/0.log" Sep 08 10:38:29.986481 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:29.986453 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b6d9c7b-xvsph_17f6623a-31b9-4352-b713-ecbb082847f6/copy-manifests/0.log" Sep 08 10:38:31.279181 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:31.279157 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c9fc4cdc-v9jpr_53096f12-27bc-4599-8c5c-a31c72bc3eee/kube-auth-proxy/0.log" Sep 08 10:38:31.306036 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:31.306008 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2m56t_48567896-0f21-404c-863a-1ece4631e205/istio-proxy/0.log" Sep 08 10:38:31.322050 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:31.322020 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2m56t_48567896-0f21-404c-863a-1ece4631e205/istio-proxy/1.log" Sep 08 10:38:31.358852 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:31.358819 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-qljfd_e6596df5-5301-45a5-9116-dc2a4268bc80/payload-pre-processing/0.log" Sep 08 10:38:33.389488 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:33.389459 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c9fc4cdc-v9jpr_53096f12-27bc-4599-8c5c-a31c72bc3eee/kube-auth-proxy/0.log" Sep 08 10:38:33.416487 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:33.416457 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2m56t_48567896-0f21-404c-863a-1ece4631e205/istio-proxy/0.log" Sep 08 10:38:33.431840 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:33.431811 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2m56t_48567896-0f21-404c-863a-1ece4631e205/istio-proxy/1.log" Sep 08 10:38:33.466760 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:33.466728 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-qljfd_e6596df5-5301-45a5-9116-dc2a4268bc80/payload-pre-processing/0.log" Sep 08 10:38:38.129274 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:38.127936 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-28d92_ce8608f0-a21a-45fb-affd-0857e3802aca/kube-multus/0.log" Sep 08 10:38:38.376310 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:38.376281 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5mv87_3b9c86ed-68b3-4864-9824-8e55025d48ba/kube-multus-additional-cni-plugins/0.log" Sep 08 10:38:38.398993 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:38.398957 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5mv87_3b9c86ed-68b3-4864-9824-8e55025d48ba/egress-router-binary-copy/0.log" Sep 08 10:38:38.421284 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:38.421260 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5mv87_3b9c86ed-68b3-4864-9824-8e55025d48ba/cni-plugins/0.log" Sep 08 10:38:38.443316 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:38.443294 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5mv87_3b9c86ed-68b3-4864-9824-8e55025d48ba/bond-cni-plugin/0.log" Sep 08 10:38:38.467369 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:38.467346 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5mv87_3b9c86ed-68b3-4864-9824-8e55025d48ba/routeoverride-cni/0.log" Sep 08 10:38:38.496133 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:38.496110 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5mv87_3b9c86ed-68b3-4864-9824-8e55025d48ba/whereabouts-cni-bincopy/0.log" Sep 08 10:38:38.523396 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:38.523374 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5mv87_3b9c86ed-68b3-4864-9824-8e55025d48ba/whereabouts-cni/0.log" Sep 08 10:38:38.878547 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:38.878466 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jgstq_01480054-d16a-4820-af73-7a493b53ead0/network-metrics-daemon/0.log" Sep 08 10:38:38.897954 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:38.897929 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jgstq_01480054-d16a-4820-af73-7a493b53ead0/kube-rbac-proxy/0.log" Sep 08 10:38:39.807514 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:39.807482 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fcgv_1a73b204-193b-48be-9f93-a6cf774ec172/ovn-controller/0.log" Sep 08 10:38:39.826496 ip-10-0-132-206 kubenswrapper[2576]: I0908 10:38:39.826471 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fcgv_1a73b204-193b-48be-9f93-a6cf774ec172/ovn-acl-logging/0.log"